Feb 16 19:39:57.299461 np0000155677 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. Feb 16 19:39:58.634333 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.common.config [-] Logging enabled! Feb 16 19:39:58.634948 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.0.0.0b3.dev139 Feb 16 19:39:58.634948 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=104694) setup_logging /opt/stack/neutron/neutron/common/config.py:124}} Feb 16 19:39:58.635077 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... Feb 16 19:39:58.635161 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=104694) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} Feb 16 19:39:58.635351 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=104694) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} Feb 16 19:39:58.635453 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=104694) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} Feb 16 19:39:58.635514 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=104694) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} Feb 16 19:39:58.635575 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=104694) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} Feb 16 19:39:58.635726 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [-] Loading core plugin: ml2 Feb 16 19:39:58.926531 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.926531 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.926882 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.927232 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.927313 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.927367 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.966675 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.966675 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.966950 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.973510 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.973510 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:58.973755 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.184692 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.184803 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.184846 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.185277 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.185455 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.185455 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.248320 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.248320 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.248584 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.248584 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.248725 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301588 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301588 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301588 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301588 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301588 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301936 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301936 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301936 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301936 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.301936 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302269 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302269 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302269 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302269 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302269 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302521 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302521 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302521 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302521 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302521 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302820 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302820 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302820 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.302820 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.303044 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:39:59.303044 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:39:59.303044 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:39:59.303256 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:39:59.303256 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:39:59.303256 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:39:59.303506 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] Feb 16 19:39:59.304214 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] Feb 16 19:39:59.304808 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete Feb 16 19:39:59.305314 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) Feb 16 19:39:59.305396 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] Feb 16 19:39:59.305584 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) Feb 16 19:39:59.305721 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['geneve'] Feb 16 19:39:59.305897 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] Feb 16 19:39:59.306399 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] Feb 16 19:39:59.306482 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] Feb 16 19:39:59.306567 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] Feb 16 19:39:59.343989 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] Feb 16 19:39:59.344160 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] Feb 16 19:39:59.344160 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. Feb 16 19:39:59.345165 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.345234 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:39:59.373740 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' Feb 16 19:39:59.373740 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete Feb 16 19:39:59.374090 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' Feb 16 19:39:59.374090 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] Feb 16 19:39:59.382898 np0000155677 neutron-ovn-maintenance-worker[104694]: WARNING oslo_policy.policy [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. Feb 16 19:39:59.383118 np0000155677 neutron-ovn-maintenance-worker[104694]: WARNING oslo_policy.policy [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. Feb 16 19:39:59.423667 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO dbcounter [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Registered counter for database neutron Feb 16 19:39:59.438991 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG oslo_db.sqlalchemy.engines [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=104694) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} Feb 16 19:39:59.854529 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG dbcounter [-] [104694] Writer thread running {{(pid=104694) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Feb 16 19:40:00.044609 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Initializing driver for type 'local' Feb 16 19:40:00.044834 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Initializing driver for type 'vlan' Feb 16 19:40:00.158459 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] VlanTypeDriver initialization complete Feb 16 19:40:00.158574 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Initializing extension driver 'port_security' Feb 16 19:40:00.158637 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.extensions.port_security [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] PortSecurityExtensionDriver initialization complete Feb 16 19:40:00.158733 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Initializing extension driver 'qos' Feb 16 19:40:00.158797 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.plugins.ml2.extensions.qos [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QosExtensionDriver initialization complete {{(pid=104694) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} Feb 16 19:40:00.158877 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Initializing mechanism driver 'ovn' Feb 16 19:40:00.158961 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Starting OVNMechanismDriver Feb 16 19:40:00.160543 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > process before_spawn 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.160633 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > process after_init 55550000, True {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.160748 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > segment after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.160824 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > segment after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.160912 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > segment after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.160984 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161084 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161168 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161243 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161339 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161412 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161501 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161579 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161654 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161759 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > address_group after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161828 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > address_group after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.161913 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > address_group after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.162100 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.162251 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.162440 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.162837 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.services.logapi.drivers.ovn.driver [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] OVN logging driver registered Feb 16 19:40:00.164016 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.plugins.ml2.plugin [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=104694) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Feb 16 19:40:00.164237 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > port after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > port after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164428 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > port after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164513 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > subnet after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164596 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > subnet after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164596 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > subnet after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164790 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > network after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164790 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > network after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164948 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > network after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.164948 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165103 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165103 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165269 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165269 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165405 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165405 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > address_group after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165537 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > address_group after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165537 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > address_group after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.165723 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ML2 OVO RPC backend initialized. {{(pid=104694) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} Feb 16 19:40:00.166292 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.quota [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded quota_driver: . Feb 16 19:40:00.166438 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.plugin [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Modular L2 Plugin initialization complete Feb 16 19:40:00.166438 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading services supported by the core plugin {{(pid=104694) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:169}} Feb 16 19:40:00.166620 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.plugins.ml2.managers [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Got port-security extension from driver 'port_security' Feb 16 19:40:00.166737 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=104694) _load_service_plugins /opt/stack/neutron/neutron/manager.py:194}} Feb 16 19:40:00.166822 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading Plugin: ovn-router Feb 16 19:40:00.173860 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.173946 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: port after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174045 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: port after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174109 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: subnet after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174194 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: subnetpool_address_scope after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174276 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: port before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174354 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: port after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174434 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > port precommit_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174515 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: port before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174593 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: port before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174673 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174771 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174852 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.174939 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: subnet after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.175502 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Tracking information for resource: router configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:40:00.175570 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.quota.resource_registry [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Tracking information for resource: floatingip configured {{(pid=104694) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Feb 16 19:40:00.175620 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.services.ovn_l3.plugin [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Starting OVNL3RouterPlugin Feb 16 19:40:00.175831 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.175994 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router before_create 55550100, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.176069 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_delete 55550100, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.176207 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_create 55550100, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.176283 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_update 55550100, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.178719 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.services.provider_configuration [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Service providers = [] {{(pid=104694) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:171}} Feb 16 19:40:00.178841 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.db.servicetype_db [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=104694) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:42}} Feb 16 19:40:00.179819 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_create 55549900, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.179946 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router_interface after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.179946 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > floatingip after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180088 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180088 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180281 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180281 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180402 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > floatingip after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180402 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180529 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180589 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router after_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180655 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180740 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router after_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180829 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > router after_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.180913 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.services.service_base [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=104694) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} Feb 16 19:40:00.181012 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > process after_init 55550001, True {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.181102 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=104694) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:229}} Feb 16 19:40:00.181181 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading Plugin: auto_allocate Feb 16 19:40:00.181880 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: network precommit_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.181972 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: network precommit_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.182097 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=104694) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:229}} Feb 16 19:40:00.182179 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading Plugin: tag Feb 16 19:40:00.184499 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=104694) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:229}} Feb 16 19:40:00.184612 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading Plugin: timestamp Feb 16 19:40:00.185548 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=104694) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:229}} Feb 16 19:40:00.185614 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading Plugin: network_ip_availability Feb 16 19:40:00.187281 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.187365 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.187453 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron_lib.callbacks.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104694) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.187712 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=104694) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:229}} Feb 16 19:40:00.187795 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading Plugin: flavors Feb 16 19:40:00.188456 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=104694) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:229}} Feb 16 19:40:00.188521 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading Plugin: revisions Feb 16 19:40:00.189447 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.manager [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=104694) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:229}} Feb 16 19:40:00.189555 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG oslo_concurrency.lockutils [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.554s {{(pid=104694) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} Feb 16 19:40:00.189716 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=104694) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} Feb 16 19:40:00.189802 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Initializing extension manager. Feb 16 19:40:00.190024 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: __init__.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.190083 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: __pycache__ {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.190171 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: address_group.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.190435 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.190630 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: address-group Feb 16 19:40:00.190715 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: address_scope.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.190952 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.191107 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: address-scope Feb 16 19:40:00.191180 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: admin_state_down_before_update.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.191364 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.191510 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins Feb 16 19:40:00.191581 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: agent.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.191837 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.191977 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: agent Feb 16 19:40:00.192049 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: agent_resources_synced.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.192275 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.192410 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension agent-resources-synced not supported by any of loaded plugins Feb 16 19:40:00.192481 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: allowedaddresspairs.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.193026 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.193179 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: allowed-address-pairs Feb 16 19:40:00.193246 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: auto_allocated_topology.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.193450 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.193597 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: auto-allocated-topology Feb 16 19:40:00.193660 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: availability_zone.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.193907 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.194047 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: availability_zone Feb 16 19:40:00.194119 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: availability_zone_filter.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.194304 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.194437 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension availability_zone_filter not supported by any of loaded plugins Feb 16 19:40:00.194507 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: data_plane_status.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.194685 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.194845 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension data-plane-status not supported by any of loaded plugins Feb 16 19:40:00.194915 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: default_subnetpools.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.195095 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.195235 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: default-subnetpools Feb 16 19:40:00.195307 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: dhcpagentscheduler.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.195596 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.195762 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: dhcp_agent_scheduler Feb 16 19:40:00.195824 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: dns.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.196009 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.196146 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: dns-integration Feb 16 19:40:00.196235 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: dns_domain_ports.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.196416 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.196551 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: dns-domain-ports Feb 16 19:40:00.196621 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.196824 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.196958 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: dns-integration-domain-keywords Feb 16 19:40:00.197027 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: dvr.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.197228 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.197369 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension dvr not supported by any of loaded plugins Feb 16 19:40:00.197440 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: ecmp_routes.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.197616 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.197777 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension ecmp_routes not supported by any of loaded plugins Feb 16 19:40:00.197845 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: empty_string_filtering.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.198025 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.198169 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: empty-string-filtering Feb 16 19:40:00.198238 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.198415 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.198555 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins Feb 16 19:40:00.198606 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.198813 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.198950 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins Feb 16 19:40:00.199032 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: external_net.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.199196 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.199332 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: external-net Feb 16 19:40:00.199402 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: extra_dhcp_opt.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.199576 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.199726 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: extra_dhcp_opt Feb 16 19:40:00.199799 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: extraroute.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.199972 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.200109 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: extraroute Feb 16 19:40:00.200208 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: extraroute_atomic.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.200395 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.200535 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: extraroute-atomic Feb 16 19:40:00.200591 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: filter_validation.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.200807 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.200943 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: filter-validation Feb 16 19:40:00.201013 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: fip_pf_description.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.201192 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.201326 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins Feb 16 19:40:00.201396 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: fip_pf_detail.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.201570 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.201725 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: floating-ip-port-forwarding-detail Feb 16 19:40:00.201796 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: fip_pf_port_range.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.201968 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.202104 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins Feb 16 19:40:00.202173 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: fip_port_details.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.202346 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.202482 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: fip-port-details Feb 16 19:40:00.202551 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: flavors.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.202759 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.202896 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: flavors Feb 16 19:40:00.202966 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.203310 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.203455 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins Feb 16 19:40:00.203527 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: floatingip_pools.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.203760 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.203901 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: floatingip-pools Feb 16 19:40:00.203966 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: ip_allocation.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.204151 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.204306 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension ip_allocation not supported by any of loaded plugins Feb 16 19:40:00.204377 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: ip_substring_port_filtering.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.204554 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.204699 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension ip-substring-filtering not supported by any of loaded plugins Feb 16 19:40:00.204779 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l2_adjacency.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.204953 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.205092 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension l2_adjacency not supported by any of loaded plugins Feb 16 19:40:00.205162 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.205479 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.205626 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: router Feb 16 19:40:00.205706 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_conntrack_helper.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.206178 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.206327 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension l3-conntrack-helper not supported by any of loaded plugins Feb 16 19:40:00.206400 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.206589 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.206746 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: enable-default-route-bfd Feb 16 19:40:00.206817 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.207008 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.207142 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: enable-default-route-ecmp Feb 16 19:40:00.207211 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_ext_gw_mode.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.207385 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.207523 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: ext-gw-mode Feb 16 19:40:00.207592 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_ext_ha_mode.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.207790 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.208516 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: l3-ha Feb 16 19:40:00.208516 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.208516 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.208516 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins Feb 16 19:40:00.208516 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_extra_gws.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.208655 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.208771 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: external-gateway-multihoming Feb 16 19:40:00.208866 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_flavors.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.209102 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.209233 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: l3-flavors Feb 16 19:40:00.209311 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_ndp_proxy.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.209511 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.209637 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension l3-ndp-proxy not supported by any of loaded plugins Feb 16 19:40:00.209720 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.209900 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.210036 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins Feb 16 19:40:00.210104 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: l3agentscheduler.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.210523 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.210663 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension l3_agent_scheduler not supported by any of loaded plugins Feb 16 19:40:00.210749 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: local_ip.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.211005 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.211148 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension local_ip not supported by any of loaded plugins Feb 16 19:40:00.211219 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: logging.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.211454 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.211586 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension logging not supported by any of loaded plugins Feb 16 19:40:00.211659 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: metering.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.211926 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.212055 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension metering not supported by any of loaded plugins Feb 16 19:40:00.212122 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.212326 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.212463 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins Feb 16 19:40:00.212551 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: multiprovidernet.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.212717 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.212869 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: multi-provider Feb 16 19:40:00.212947 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: netmtu.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.213114 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.213247 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: net-mtu Feb 16 19:40:00.213317 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: netmtu_writable.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.213493 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.213787 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: net-mtu-writable Feb 16 19:40:00.213858 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: network_availability_zone.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.214068 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.214217 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: network_availability_zone Feb 16 19:40:00.214452 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: network_ha.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.214635 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.214793 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension network_ha not supported by any of loaded plugins Feb 16 19:40:00.214853 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: network_ip_availability.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.215336 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.215336 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: network-ip-availability Feb 16 19:40:00.215336 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: network_ip_availability_details.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.215460 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.215584 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: network-ip-availability-details Feb 16 19:40:00.215637 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: network_segment_range.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.215915 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.216061 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension network-segment-range not supported by any of loaded plugins Feb 16 19:40:00.216119 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: pagination.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.216350 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.216425 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: pagination Feb 16 19:40:00.216478 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_device_profile.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.216650 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.216815 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: port-device-profile Feb 16 19:40:00.217120 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_hardware_offload_type.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.217120 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.217202 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-hardware-offload-type not supported by any of loaded plugins Feb 16 19:40:00.217245 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.217425 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.217559 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins Feb 16 19:40:00.217632 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_hints.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.217827 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.217964 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-hints not supported by any of loaded plugins Feb 16 19:40:00.218030 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_mac_address_override.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.218212 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.218567 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-mac-override not supported by any of loaded plugins Feb 16 19:40:00.218567 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_mac_address_regenerate.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.218567 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.218738 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: port-mac-address-regenerate Feb 16 19:40:00.218792 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_numa_affinity_policy.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.218983 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.219119 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: port-numa-affinity-policy Feb 16 19:40:00.219181 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.219362 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.219494 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: port-numa-affinity-policy-socket Feb 16 19:40:00.219564 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_resource_request.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.219761 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.219891 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-resource-request not supported by any of loaded plugins Feb 16 19:40:00.220163 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_resource_request_groups.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.220163 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.220276 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-resource-request-groups not supported by any of loaded plugins Feb 16 19:40:00.220355 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: port_trusted_vif.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.220527 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.220661 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-trusted-vif not supported by any of loaded plugins Feb 16 19:40:00.220746 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: portbindings.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.221271 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.221271 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: binding Feb 16 19:40:00.221271 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: portbindings_extended.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.221395 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: binding-extended Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: portsecurity.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: port-security Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: project_id.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: project-id Feb 16 19:40:00.222188 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: providernet.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.222494 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.222494 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: provider Feb 16 19:40:00.222570 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qinq.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.222765 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x8a88 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.222903 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: qinq Feb 16 19:40:00.222974 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.223659 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.223820 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos not supported by any of loaded plugins Feb 16 19:40:00.223889 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_bw_limit_direction.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.224104 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.224276 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-bw-limit-direction not supported by any of loaded plugins Feb 16 19:40:00.224336 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.224524 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.224656 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins Feb 16 19:40:00.224748 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_default.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.224964 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.225117 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-default not supported by any of loaded plugins Feb 16 19:40:00.225189 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_fip.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.225371 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.225510 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-fip not supported by any of loaded plugins Feb 16 19:40:00.225583 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_fip_network_policy.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.225779 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.225916 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-fip-network-policy not supported by any of loaded plugins Feb 16 19:40:00.226004 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_gateway_ip.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.226198 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.226340 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-gateway-ip not supported by any of loaded plugins Feb 16 19:40:00.226413 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_port_network_policy.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.226617 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.226782 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-port-network-policy not supported by any of loaded plugins Feb 16 19:40:00.226855 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.227065 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.227221 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-pps-minimum not supported by any of loaded plugins Feb 16 19:40:00.227293 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.227484 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.227625 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins Feb 16 19:40:00.227711 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_pps_rule.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.227933 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.228074 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-pps not supported by any of loaded plugins Feb 16 19:40:00.228150 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_rule_type_details.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.228365 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.228508 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-rule-type-details not supported by any of loaded plugins Feb 16 19:40:00.228579 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_rule_type_filter.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.228773 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.228911 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-rule-type-filter not supported by any of loaded plugins Feb 16 19:40:00.228983 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: qos_rules_alias.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.229173 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.229309 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension qos-rules-alias not supported by any of loaded plugins Feb 16 19:40:00.229380 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: quota_check_limit.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.229558 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.229712 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: quota-check-limit Feb 16 19:40:00.229804 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: quota_check_limit_default.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.230008 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.230150 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: quota-check-limit-default Feb 16 19:40:00.230220 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: quotasv2.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.230613 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.230678 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: quotas Feb 16 19:40:00.230778 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: quotasv2_detail.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.231049 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.231127 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: quota_details Feb 16 19:40:00.231198 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: rbac.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.231748 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control tenant access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.231907 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: rbac-policies Feb 16 19:40:00.231981 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: rbac_address_group.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.232182 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.232399 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension rbac-address-group not supported by any of loaded plugins Feb 16 19:40:00.232497 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: rbac_address_scope.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.232744 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.232901 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: rbac-address-scope Feb 16 19:40:00.232961 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: rbac_security_groups.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.233154 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.233307 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: rbac-security-groups Feb 16 19:40:00.233386 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: rbac_subnetpool.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.233547 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.233682 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension rbac-subnetpool not supported by any of loaded plugins Feb 16 19:40:00.233799 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: revisionifmatch.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.233987 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.234130 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: revision-if-match Feb 16 19:40:00.234204 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: revisions.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.234401 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.234542 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: standard-attr-revisions Feb 16 19:40:00.234615 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: router_availability_zone.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.234844 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.234988 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: router_availability_zone Feb 16 19:40:00.235061 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: router_enable_snat.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.235238 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.235379 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: router-enable-snat Feb 16 19:40:00.235451 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: routerservicetype.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.235630 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.235796 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension router-service-type not supported by any of loaded plugins Feb 16 19:40:00.235872 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: security_groups_default_rules.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.236241 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.236385 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: security-groups-default-rules Feb 16 19:40:00.236458 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.236649 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.236811 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: security-groups-normalized-cidr Feb 16 19:40:00.236882 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: security_groups_port_filtering.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.237072 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.237208 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension port-security-groups-filtering not supported by any of loaded plugins Feb 16 19:40:00.237281 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: security_groups_remote_address_group.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.237466 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.237621 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: security-groups-remote-address-group Feb 16 19:40:00.237721 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.237923 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.238067 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: security-groups-rules-belongs-to-default-sg Feb 16 19:40:00.238139 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: security_groups_shared_filtering.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.238319 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.238460 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: security-groups-shared-filtering Feb 16 19:40:00.238530 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: securitygroup.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.239990 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.240139 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: security-group Feb 16 19:40:00.240237 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: segment.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.240569 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.240740 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension segment not supported by any of loaded plugins Feb 16 19:40:00.240811 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.241010 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.241153 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins Feb 16 19:40:00.241223 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: servicetype.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.241423 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.241590 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: service-type Feb 16 19:40:00.241666 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: sorting.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.241895 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.241972 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: sorting Feb 16 19:40:00.242043 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: standard_attr_segment.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.242230 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.242373 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension standard-attr-segment not supported by any of loaded plugins Feb 16 19:40:00.242445 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: standardattrdescription.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.242644 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.242809 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: standard-attr-description Feb 16 19:40:00.242879 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: stateful_security_group.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.243061 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.243200 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: stateful-security-group Feb 16 19:40:00.243273 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: stdattrs_common.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.243434 np0000155677 neutron-ovn-maintenance-worker[104694]: WARNING neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py Feb 16 19:40:00.243507 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.243683 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.243843 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: subnet-dns-publish-fixed-ip Feb 16 19:40:00.243916 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: subnet_external_network.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.244146 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.244278 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: subnet-external-network Feb 16 19:40:00.244358 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: subnet_onboard.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.244559 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.244705 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension subnet_onboard not supported by any of loaded plugins Feb 16 19:40:00.244797 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: subnet_segmentid_writable.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.244978 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.245120 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension subnet-segmentid-writable not supported by any of loaded plugins Feb 16 19:40:00.245194 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: subnet_service_types.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.245372 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.245515 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: subnet-service-types Feb 16 19:40:00.245589 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: subnetallocation.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.245802 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.245943 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: subnet_allocation Feb 16 19:40:00.246016 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.246217 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.246358 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: subnetpool-prefix-ops Feb 16 19:40:00.246430 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: tag_creation.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.246758 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.246973 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: tag-creation Feb 16 19:40:00.246973 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.247161 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.247306 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins Feb 16 19:40:00.247376 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: tagging.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.248056 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.248206 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: standard-attr-tag Feb 16 19:40:00.248300 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: timestamp.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.248514 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.248656 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: standard-attr-timestamp Feb 16 19:40:00.248750 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: trunk.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.248941 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.249079 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension trunk not supported by any of loaded plugins Feb 16 19:40:00.249152 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: trunk_details.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.249414 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.249612 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension trunk-details not supported by any of loaded plugins Feb 16 19:40:00.249726 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: uplink_status_propagation.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.249986 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.250178 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension uplink-status-propagation not supported by any of loaded plugins Feb 16 19:40:00.250277 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.250527 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.250746 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins Feb 16 19:40:00.250849 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loading extension file: vlantransparent.py {{(pid=104694) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Feb 16 19:40:00.251138 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=104694) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Feb 16 19:40:00.251361 np0000155677 neutron-ovn-maintenance-worker[104694]: INFO neutron.api.extensions [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Loaded extension: vlan-transparent Feb 16 19:40:00.253863 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Full set of CONF: {{(pid=104694) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} Feb 16 19:40:00.253931 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ******************************************************************************** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2804}} Feb 16 19:40:00.254035 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Configuration options gathered from: {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2805}} Feb 16 19:40:00.254097 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2806}} Feb 16 19:40:00.254229 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2807}} Feb 16 19:40:00.254299 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ================================================================================ {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2809}} Feb 16 19:40:00.254452 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] agent_down_time = 75 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.254553 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] allow_automatic_dhcp_failover = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.254679 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] allow_bulk = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.254903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255003 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] api_extensions_path = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255112 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] api_paste_config = api-paste.ini {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255220 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] api_workers = 4 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255394 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] auth_strategy = keystone {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255436 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] backlog = 4096 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255516 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] base_mac = fa:16:3e:00:00:00 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255646 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] bind_host = 0.0.0.0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255765 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] bind_port = 9696 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255872 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] client_socket_timeout = 900 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.255970 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] config_dir = [] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.256059 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.256280 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] config_source = [] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.256280 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] control_exchange = neutron {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.256380 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] core_plugin = ml2 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.256473 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] debug = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.256576 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] default_availability_zones = [] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.256744 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257007 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] dhcp_agent_notification = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257007 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] dhcp_agents_per_network = 1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257079 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] dhcp_lease_duration = 86400 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257144 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] dhcp_load_type = networks {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257234 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] dns_domain = openstacklocal {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257333 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257429 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] enable_default_route_bfd = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257522 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] enable_default_route_ecmp = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257621 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] enable_dvr = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257726 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] enable_new_agents = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257830 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] enable_services_on_agents_with_admin_state_down = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.257920 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] enable_snat_by_default = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258004 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] enable_traditional_dhcp = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258086 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] external_dns_driver = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258172 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] filter_validation = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258263 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] global_physnet_mtu = 1430 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258329 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] graceful_shutdown_timeout = 60 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258463 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] host = np0000155677 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258552 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] host_dvr_for_dhcp = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258658 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] http_retries = 3 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258841 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] instance_format = [instance: %(uuid)s] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258884 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.258970 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ipam_driver = internal {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259056 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ipv6_pd_enabled = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259149 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] l3_ha = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259247 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259336 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] l3_ha_network_physical_name = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259427 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] l3_ha_network_type = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259532 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_color = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259624 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_config_append = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259745 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259856 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_dir = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.259948 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_file = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260013 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_options = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260099 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_rotate_interval = 1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260189 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_rotate_interval_type = days {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260316 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] log_rotation_type = none {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260396 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260463 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260568 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260666 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260751 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260853 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_allowed_address_pair = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.260945 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_dns_nameservers = 5 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.261048 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_header_line = 16384 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.261138 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_l3_agents_per_router = 3 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.261270 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_logfile_count = 30 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.261422 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_logfile_size_mb = 200 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.261549 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_routes = 30 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.261666 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] max_subnet_host_routes = 20 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262047 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] my_ip = 199.204.45.53 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262047 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] my_ipv6 = 2604:e100:1:0:f816:3eff:fe10:68f4 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262121 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] network_auto_schedule = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262230 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] network_link_prefix = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262376 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262511 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] notify_nova_on_port_data_changes = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262668 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] notify_nova_on_port_status_changes = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.262869 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] pagination_max_limit = -1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263018 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] periodic_fuzzy_delay = 5 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263148 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] periodic_interval = 40 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263493 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] publish_errors = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263493 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rate_limit_burst = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263493 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rate_limit_except_level = CRITICAL {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263642 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rate_limit_interval = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263798 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] retry_until_window = 30 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.263944 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] router_distributed = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.264075 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rpc_resources_processing_step = 20 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.264199 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rpc_response_max_timeout = 600 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.264397 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rpc_response_timeout = 60 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.264530 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rpc_state_report_workers = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.264664 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] rpc_workers = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.264817 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] send_events_interval = 2 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.264963 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] service_plugins = ['ovn-router'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265111 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] setproctitle = on {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265216 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] shell_completion = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265303 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] state_path = /opt/stack/data/neutron {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265393 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] syslog_log_facility = LOG_USER {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265490 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] tcp_keepidle = 600 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265591 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] transport_url = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265676 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] use_journal = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265780 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] use_json = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.265920 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] use_ssl = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266290 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] use_stderr = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266290 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] use_syslog = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266290 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] vlan_qinq = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266290 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] vlan_transparent = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266393 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] wsgi_default_pool_size = 100 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266514 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] wsgi_keep_alive = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266546 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266625 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] wsgi_server_debug = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.266761 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_concurrency.disable_process_locking = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.266874 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.266978 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.connection_string = messaging:// {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267068 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.enabled = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267159 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.es_doc_type = notification {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267247 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.es_scroll_size = 10000 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267337 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.es_scroll_time = 2m {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267425 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.filter_error_trace = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267520 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.hmac_keys = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267601 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.sentinel_service_name = mymaster {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267719 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.socket_timeout = 0.1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267810 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.trace_requests = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267898 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler.trace_sqlalchemy = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.267992 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler_jaeger.process_tags = {} {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268078 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler_jaeger.service_name_prefix = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268167 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] profiler_otlp.service_name_prefix = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268336 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268450 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268450 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_metrics.metrics_process_name = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268565 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268637 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268762 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.enforce_new_defaults = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.268930 np0000155677 neutron-ovn-maintenance-worker[104694]: WARNING oslo_config.cfg [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. Feb 16 19:40:00.269025 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.enforce_scope = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269124 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.policy_default_rule = default {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269219 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269310 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269407 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269507 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269600 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269684 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269812 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.269935 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_policy.remote_timeout = 60.0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270426 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270426 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] service_providers.service_provider = [] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270426 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.auth_section = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270426 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.auth_type = password {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270426 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.auth_url = https://199.204.45.53/identity {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270426 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.cafile = /opt/stack/data/ca-bundle.pem {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270959 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.certfile = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270959 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.collect_timing = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270959 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.default_domain_id = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270959 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.default_domain_name = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270959 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.domain_id = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.270959 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.domain_name = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.271903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.endpoint_type = public {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.271903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.insecure = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.271903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.keyfile = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.271903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.password = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.271903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.project_domain_id = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.271903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.project_domain_name = Default {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.271903 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.project_id = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.project_name = service {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.region_name = RegionOne {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.split_loggers = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.system_scope = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.timeout = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.trust_id = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272342 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.user_domain_id = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272756 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.user_domain_name = Default {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272756 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.user_id = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272756 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] nova.username = nova {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272756 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.auth_section = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272756 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.auth_type = password {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272756 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.cafile = /opt/stack/data/ca-bundle.pem {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.272756 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.certfile = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273179 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.collect_timing = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273179 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.endpoint_type = public {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273179 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.insecure = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273179 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.keyfile = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273179 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.region_name = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273457 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.split_loggers = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273457 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] placement.timeout = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273457 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.auth_section = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273457 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.auth_type = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273673 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.cafile = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273673 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.certfile = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273673 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.collect_timing = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273902 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.connect_retries = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.273902 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.connect_retry_delay = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274017 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.enable_notifications = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274017 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.endpoint_override = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274123 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.insecure = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274180 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.interface = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274243 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.keyfile = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274328 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.max_version = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274407 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.min_version = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274465 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.region_name = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274583 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.retriable_status_codes = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274655 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.service_name = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274758 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.service_type = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274829 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.split_loggers = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274895 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.status_code_retries = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.274957 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.status_code_retry_delay = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275049 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.timeout = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275117 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.valid_interfaces = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275184 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ironic.version = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275289 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] cli_script.dry_run = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275377 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275455 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275525 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275621 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275852 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275852 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.275852 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276028 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276028 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276146 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.hostname = np0000155677 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276210 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276299 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276364 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276466 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276537 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276639 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276728 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276795 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.276890 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277079 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277079 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277219 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277219 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277354 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277354 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277482 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277544 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277606 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277685 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277796 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277861 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.ssl = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.277924 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278002 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278072 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278175 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278240 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.ssl_version = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278329 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278410 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_notifications.driver = [] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278493 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_notifications.retry = -1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278592 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278669 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_messaging_notifications.transport_url = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278772 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_reports.file_event_handler = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278855 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_reports.file_event_handler_interval = 1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.278937 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_reports.log_dir = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279023 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279110 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.default_quota = -1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279196 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279279 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_floatingip = 50 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279361 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_network = 100 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279445 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_port = 500 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279528 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_rbac_policy = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279612 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_router = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279707 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_security_group = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279800 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_security_group_rule = 100 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279883 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.quota_subnet = 100 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.279967 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] QUOTAS.track_quota_usage = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280060 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280145 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep.group = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280227 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep.helper_command = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280346 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep.log_daemon_traceback = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280434 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep.logger_name = oslo_privsep.daemon {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280518 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep.thread_pool_size = 16 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280601 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep.user = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280690 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280792 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_dhcp_release.group = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280873 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_dhcp_release.helper_command = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.280956 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281053 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281122 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_dhcp_release.thread_pool_size = 16 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281204 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_dhcp_release.user = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281291 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281373 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_ovs_vsctl.group = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281454 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_ovs_vsctl.helper_command = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281537 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281621 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281722 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281803 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_ovs_vsctl.user = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281889 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_namespace.capabilities = [21] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.281971 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_namespace.group = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282052 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_namespace.helper_command = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282133 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_namespace.log_daemon_traceback = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282218 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282300 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_namespace.thread_pool_size = 16 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282381 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_namespace.user = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282467 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_conntrack.capabilities = [12] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282549 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_conntrack.group = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282629 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_conntrack.helper_command = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282729 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_conntrack.log_daemon_traceback = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282813 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282895 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_conntrack.thread_pool_size = 16 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.282976 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_conntrack.user = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283066 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_link.capabilities = [12, 21] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283146 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_link.group = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283227 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_link.helper_command = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283310 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_link.log_daemon_traceback = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283395 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283477 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_link.thread_pool_size = 16 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283557 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] privsep_link.user = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283651 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283760 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283843 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] AGENT.use_helper_for_ns_read = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.283930 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] SECURITYGROUP.enable_ipset = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284018 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] SECURITYGROUP.enable_security_group = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284100 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] SECURITYGROUP.firewall_driver = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284187 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284288 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] experimental.ipv6_pd_enabled = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284388 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284472 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.external_network_type = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284562 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.mechanism_drivers = ['ovn'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284653 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.overlay_ip_version = 4 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284755 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.path_mtu = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284841 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.physical_network_mtus = {} {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.284929 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.tenant_network_types = ['geneve'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285018 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285110 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285203 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2_type_flat.flat_networks = ['public'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285293 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2_type_geneve.max_header_size = 38 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285382 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285472 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285561 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] network_log.burst_limit = 25 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285641 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] network_log.local_output_log_base = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285739 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] network_log.rate_limit = 100 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285828 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.asyncio_connection = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285909 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.asyncio_slave_connection = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.285995 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.backend = sqlalchemy {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286082 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.connection = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286166 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.connection_debug = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286250 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.connection_parameters = {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286333 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.connection_recycle_time = 3600 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286413 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.connection_trace = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286495 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.db_inc_retry_interval = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286577 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.db_max_retries = 20 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286659 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.db_max_retry_interval = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286757 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.db_retry_interval = 1 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286838 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.max_overflow = 50 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.286919 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.max_pool_size = 5 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287000 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.max_retries = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287086 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.mysql_sql_mode = TRADITIONAL {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287169 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.mysql_wsrep_sync_wait = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287248 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.pool_timeout = None {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287329 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.retry_interval = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287409 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.slave_connection = **** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287491 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.sqlite_synchronous = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287572 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.synchronous_reader = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287654 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] database.use_db_reconnect = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287763 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.broadcast_arps_to_all_routers = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287845 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.dhcp_default_lease_time = 43200 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.287927 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288008 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.dns_records_ovn_owned = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288091 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.dns_servers = [] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288174 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.enable_distributed_floating_ip = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288285 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.fdb_age_threshold = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288374 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.live_migration_activation_strategy = rarp {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288455 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.localnet_learn_fdb = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288537 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.mac_binding_age_threshold = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288624 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.neutron_sync_mode = log {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288728 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_dhcp4_global_options = {} {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288812 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_dhcp6_global_options = {} {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288897 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.288981 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_metadata_enabled = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289065 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289150 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289242 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_nb_connection = ['ssl:199.204.45.53:6641'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289327 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289409 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_router_indirect_snat = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289494 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289578 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289667 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_sb_connection = ['ssl:199.204.45.53:6642'] {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289769 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289852 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovs_create_tap = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.289935 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovsdb_connection_timeout = 180 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290021 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovsdb_log_level = INFO {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290104 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovsdb_probe_interval = 60000 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290191 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290271 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.stateless_nat_enabled = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290356 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290444 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] OVS.bridge_mac_table_size = 50000 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290526 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] OVS.igmp_flood = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290607 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] OVS.igmp_flood_reports = True {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290689 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] OVS.igmp_flood_unregistered = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290794 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] OVS.igmp_snooping_enable = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290877 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] OVS.ovsdb_timeout = 10 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.290963 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.291046 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn_nb_global.ignore_lsp_down = False {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.291132 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.291197 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG cotyledon.oslo_config_glue [None req-3fcb1656-8b63-4e78-95b3-0adca0a9693e None None] ******************************************************************************** {{(pid=104694) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2828}} Feb 16 19:40:00.320884 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=105470) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} Feb 16 19:40:00.321473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2804}} Feb 16 19:40:00.321473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2805}} Feb 16 19:40:00.321473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2806}} Feb 16 19:40:00.321473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2807}} Feb 16 19:40:00.321473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2809}} Feb 16 19:40:00.321776 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.321776 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.321776 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322004 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322083 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 4 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322428 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322428 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] backlog = 4096 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322546 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322852 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] bind_host = 0.0.0.0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322852 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] bind_port = 9696 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.322852 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323196 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323196 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323358 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323613 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323613 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.323966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.324483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.324483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.324483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.324483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.324483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.324483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.324483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] host = np0000155677 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.325865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326121 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326121 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326121 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326121 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326761 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326761 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326761 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326925 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326925 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.326925 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327091 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327091 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 199.204.45.53 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327091 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = 2604:e100:1:0:f816:3eff:fe10:68f4 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327265 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327265 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327367 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327367 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327477 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327477 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327593 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327652 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327736 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327736 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327908 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.327983 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] retry_until_window = 30 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328053 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328053 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328165 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328224 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328283 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328365 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328425 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328746 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328822 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328894 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.328957 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329020 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329080 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329139 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329200 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329268 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329268 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] use_ssl = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329380 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329380 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329508 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329508 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329627 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329627 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329760 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329760 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2817}} Feb 16 19:40:00.329892 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.329953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330233 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330619 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330619 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330799 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330799 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.330799 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331479 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331479 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331479 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331479 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331479 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.331895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://199.204.45.53/identity {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = /opt/stack/data/ca-bundle.pem {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.332598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = /opt/stack/data/ca-bundle.pem {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.333880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334304 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334304 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334304 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334304 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334304 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334558 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334586 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334664 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334664 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334738 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334769 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334817 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334888 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000155677 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334917 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.334966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335041 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335105 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335139 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335169 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335203 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335235 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335449 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335478 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335523 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335565 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335604 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335633 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335708 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335769 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335882 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335910 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335951 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.335985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336015 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336075 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336109 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336164 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336212 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336243 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336302 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336416 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336463 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336493 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336567 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336677 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336740 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336784 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336821 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336863 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336902 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336941 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.336979 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337063 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337136 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337176 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337213 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337288 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337331 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 16 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337369 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337418 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337456 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337492 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337535 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 16 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337657 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337657 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337833 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337833 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337899 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337899 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337977 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.337977 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338051 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338051 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 16 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338202 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338202 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338271 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338271 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338349 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338349 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 16 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338502 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338502 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338570 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338570 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338680 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 16 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338680 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338760 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338886 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338886 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338886 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338997 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.338997 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339072 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339072 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339146 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339189 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339232 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339274 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339317 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339400 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.tenant_network_types = ['geneve'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339438 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339480 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339517 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339582 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339633 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339672 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339758 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339802 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339838 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339904 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339950 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.339988 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340025 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340070 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340110 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340147 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340189 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340228 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340265 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340308 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340393 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340436 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340484 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340527 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340565 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340651 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340689 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340770 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340770 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340850 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340850 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340919 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.340962 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341003 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341042 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341083 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341130 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341130 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341209 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341209 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341285 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341285 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341358 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341358 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341436 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341436 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341513 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341513 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341585 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:199.204.45.53:6641'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341620 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341667 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341667 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341747 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341796 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:199.204.45.53:6642'] {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341796 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341864 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341906 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341906 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341979 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.341979 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342047 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342082 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342172 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342172 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342311 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342347 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342390 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2824}} Feb 16 19:40:00.342468 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=105470) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2828}} Feb 16 19:40:00.342657 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [105470] {{(pid=105470) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} Feb 16 19:40:00.343486 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-491745', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-1257731'] for process (None), after_init {{(pid=105470) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Feb 16 19:40:00.344716 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry Feb 16 19:40:00.432970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433930 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433930 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433930 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433930 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433930 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.433930 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.436106 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.53:6641: connecting... Feb 16 19:40:00.444499 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.53:6641: connected Feb 16 19:40:00.472226 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry Feb 16 19:40:00.501339 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Feb 16 19:40:00.501430 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Feb 16 19:40:00.501478 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.501572 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=105470) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Feb 16 19:40:00.503401 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.53:6642: connecting... Feb 16 19:40:00.508797 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.53:6642: connected Feb 16 19:40:00.545308 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.545466 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.545466 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.545574 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.545574 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.545676 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.545915 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-1257958'] for port_forwarding_plugin (None), after_init {{(pid=105470) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Feb 16 19:40:00.546002 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.546069 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.546126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.546184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.546241 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.546374 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Feb 16 19:40:00.546462 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension Feb 16 19:40:00.546844 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.546939 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547297 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547365 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547417 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547470 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547523 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547573 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547636 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547727 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.547781 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.600120 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.600120 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.600316 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.600316 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.602296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.602371 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.602471 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-15502387'] for logging-plugin (None), after_init {{(pid=105470) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Feb 16 19:40:00.602596 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.602663 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:00.602757 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=105470) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Feb 16 19:40:10.051532 np0000155677 neutron-ovn-maintenance-worker[104694]: DEBUG dbcounter [-] [104694] Writing DB stats neutron:SELECT=4 {{(pid=104694) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:40:10.604056 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-02-16 19:40:10.603430 {{(pid=105470) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:97}} Feb 16 19:40:10.606466 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync port groups started @ 2026-02-16 19:40:10.605946 {{(pid=105470) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:130}} Feb 16 19:40:10.626350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writer thread running {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Feb 16 19:40:10.640811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync port groups completed @ 2026-02-16 19:40:10.640501 {{(pid=105470) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:194}} Feb 16 19:40:10.641015 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-02-16 19:40:10.640786 {{(pid=105470) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1114}} Feb 16 19:40:10.753149 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.services.segments.db [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['1e662dc7-bbd7-4f79-9523-34c3de2c8994'], 'is_dynamic': False}} {{(pid=105470) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Feb 16 19:40:10.758022 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync metadata ports started {{(pid=105470) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1059}} Feb 16 19:40:10.819842 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync metadata ports completed {{(pid=105470) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1111}} Feb 16 19:40:10.819995 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=105470) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:911}} Feb 16 19:40:11.053898 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-02-16 19:40:11.053541 {{(pid=105470) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:984}} Feb 16 19:40:11.054577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Feb 16 19:40:11.054876 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=105470) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:989}} Feb 16 19:40:11.055000 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=105470) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1048}} Feb 16 19:40:11.055082 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-02-16 19:40:11.054949 {{(pid=105470) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1283}} Feb 16 19:40:11.055147 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync ACLs started @ 2026-02-16 19:40:11.055046 {{(pid=105470) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:237}} Feb 16 19:40:11.094450 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync ACLs completed @ 2026-02-16 19:40:11.094138 {{(pid=105470) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:427}} Feb 16 19:40:11.094659 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync routers and router ports started @ 2026-02-16 19:40:11.094435 {{(pid=105470) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:585}} Feb 16 19:40:11.145170 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Feb 16 19:40:11.145505 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync routers and router ports completed @ 2026-02-16 19:40:11.145213 {{(pid=105470) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:906}} Feb 16 19:40:11.145625 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync port QoS policies started @ 2026-02-16 19:40:11.145493 {{(pid=105470) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1331}} Feb 16 19:40:11.145808 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] Starting OVNClientQosExtension Feb 16 19:40:11.161628 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync port QoS policies completed @ 2026-02-16 19:40:11.161447 {{(pid=105470) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1354}} Feb 16 19:40:11.161762 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync Floating IP QoS policies started @ 2026-02-16 19:40:11.161621 {{(pid=105470) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1359}} Feb 16 19:40:11.161837 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] Starting OVNClientQosExtension Feb 16 19:40:11.180765 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-02-16 19:40:11.180489 {{(pid=105470) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1382}} Feb 16 19:40:11.180765 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync Floating IP NAT rules started @ 2026-02-16 19:40:11.180625 {{(pid=105470) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1387}} Feb 16 19:40:11.180923 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-02-16 19:40:11.180835 {{(pid=105470) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1410}} Feb 16 19:40:11.181112 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-20fd7e3c-8a42-4399-a619-4975960a459f None None] OVN-Northbound DB sync process completed @ 2026-02-16 19:40:11.180923 {{(pid=105470) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:110}} Feb 16 19:40:20.605421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-02-16 19:40:20.605022 {{(pid=105470) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1438}} Feb 16 19:40:20.606413 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Feb 16 19:40:20.607598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] OVN-SB Sync hostname and physical networks started @ 2026-02-16 19:40:20.607219 {{(pid=105470) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1465}} Feb 16 19:40:20.611498 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.611645 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.611956 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.612409 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.612409 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.612409 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.612613 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.612613 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.612771 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.612882 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.613000 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.613904 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614014 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614073 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614117 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614188 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614269 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614344 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614399 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_qos_fip_rule_priority {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614477 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.614546 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_static_routes {{(pid=105470) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:97}} Feb 16 19:40:20.616320 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.617762 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started Feb 16 19:40:20.617882 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization Feb 16 19:40:20.618520 np0000155677 neutron-ovn-maintenance-worker[105470]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] New host np0000155677 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron Feb 16 19:40:20.626173 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.628377 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.631181 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:20.631664 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.633104 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.635755 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:20.636724 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.640635 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7537a874-f891-4b1d-ad0e-8e7897f65914 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:20.647719 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.651802 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:20.653192 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG oslo_concurrency.lockutils [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=105470) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} Feb 16 19:40:20.653444 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG oslo_concurrency.lockutils [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.000s {{(pid=105470) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} Feb 16 19:40:20.656755 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.659200 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.660283 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.662491 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Feb 16 19:40:20.665492 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.667267 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.674255 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:20.679747 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.685867 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:20.702188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.715619 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:20.720424 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.727782 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:20.732033 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.734848 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:20.735569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.750660 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:20.875805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:20.877364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:20.878317 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron_lib.callbacks.manager [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-12848', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-16321562'] for segment_host_mapping (None), after_create {{(pid=105470) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Feb 16 19:40:20.899476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:20.914011 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.914788 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.db.ovn_hash_ring_db [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Cleaned up Hash Ring nodes older than 5 days Feb 16 19:40:20.926315 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-71215643-ba0d-4bfa-b9ed-5d7e4ab2cea3 None None] Maintenance task: No inconsistencies found. Skipping {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:384}} Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:20.935483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.936723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:20.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.956917 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Maintenance task: Check missing prefix router_name in external_ids of LRPs. {{(pid=105470) update_lrouter_ports_ext_ids_name_prefix /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:501}} Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 915, in configure_nb_global Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Feb 16 19:40:20.972476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:20.973344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:20.974254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.974254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:20.985228 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:20.994903 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:20.995126 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:20.999361 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Maintenance task: Check missing prefix router_name in LRPs (took 0.01 seconds) Feb 16 19:40:21.022236 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [None req-f8dbf640-f05d-4b7c-98ea-6d171364f7c6 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.023761 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.services.segments.db [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] Segments {'baea1945-0145-4a27-8595-4a08d27c2335'} mapped to the host np0000155677 {{(pid=105470) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:21.048097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.049104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:21.066054 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_qos_fip_rule_priority' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:21.078410 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:21.080254 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [None req-dd1374e1-e051-45cc-9310-2d81f313e573 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.082377 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.087751 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched CREATE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(name=lrp-ab37a321-216d-418e-91c8-8aff5714d82e, ipv6_prefix=[], mac=fa:16:3e:5b:48:d2, gateway_chassis=[], external_ids={'neutron:is_ext_gw': 'False', 'neutron:network_name': 'neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'neutron:revision_number': '1', 'neutron:router_name': 'neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4', 'neutron:subnet_ids': '247b2a4f-27c3-48ff-8b5b-2a6e34d81c2d'}, options={}, ha_chassis_group=[], status={}, enabled=[], networks=['fd::1/64'], peer=[], ipv6_ra_configs={'address_mode': 'slaac', 'mtu': '1372', 'send_periodic': 'true'}) old= {{(pid=105470) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:21.101230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:21.102133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.103030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:21.103030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.103030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:21.116877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG oslo_concurrency.lockutils [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.416s {{(pid=105470) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} Feb 16 19:40:21.119538 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:21.120366 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_qos_fip_rule_priority' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.126937 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Feb 16 19:40:21.128949 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:21.129640 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] OVN-SB Sync hostname and physical networks completed @ 2026-02-16 19:40:21.119025 {{(pid=105470) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1492}} Feb 16 19:40:21.130391 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' {{(pid=105470) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:673}} Feb 16 19:40:21.143949 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:21.144124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.153633 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.180799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.181330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:21.184612 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [None req-9ef9f411-327e-4712-9eac-30125f89ebc1 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.187032 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [None req-a8144e35-2dc5-4674-99f4-b5dc58eb0835 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.188195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.234652 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:21.235181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7537a874-f891-4b1d-ad0e-8e7897f65914 None None] Traceback (most recent call last): Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.236438 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-7537a874-f891-4b1d-ad0e-8e7897f65914 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.251304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.252189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:21.376622 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [None req-ec300e64-e185-4b20-90d2-72dd49dcd992 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.411330 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105470) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:710}} Feb 16 19:40:21.421738 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.common.ovn.utils [-] HA Chassis Group neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994 synchronized; highest priority chassis 4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27 Feb 16 19:40:21.429017 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=HA_Chassis_Group, record=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, col_values=(('external_ids', {'neutron:availability_zone_hints': '', 'neutron:network_id': '1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'neutron:router_id': 'b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4'}),), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:21.432198 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): HAChassisGroupAddChassisCommand(_result=None, hcg=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, chassis=4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27, priority=1, columns={}) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:21.432615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.436244 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Traceback (most recent call last): Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/event.py", line 185, in notify_loop Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event match.run(event, row, updates) Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/services/ovn_l3/ovsdb_monitor.py", line 85, in run Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.l3_plugin._ovn_client.link_network_ha_chassis_group( Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2381, in link_network_ha_chassis_group Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with super().transaction(*args, **kwargs) as t: Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self.create_transaction( Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.result = self.commit() Feb 16 19:40:21.443342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^ Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise result.ex Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event txn.results.put(txn.do_commit()) Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^^^ Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise RuntimeError(msg) Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:21.444131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Feb 16 19:40:21.602558 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Feb 16 19:40:21.603077 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-365da60d-b638-4afe-b851-4eab09c8f7e0 None None] OVN-Southbound DB sync process completed @ 2026-02-16 19:40:21.602722 {{(pid=105470) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1446}} Feb 16 19:40:22.715027 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched DELETE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(name=lrp-a3b35ad3-6c89-4653-b097-0f605244fd26, ipv6_prefix=[], mac=fa:16:3e:b5:cd:f1, gateway_chassis=[], external_ids={'neutron:is_ext_gw': 'True', 'neutron:network_name': 'neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071', 'neutron:revision_number': '1', 'neutron:router_name': 'neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4', 'neutron:subnet_ids': '0ee58c77-b622-4ab4-adb2-400b1ab81596'}, options={'gateway_mtu': '1372', 'reside-on-redirect-chassis': 'true'}, ha_chassis_group=[], status={'hosting-chassis': '4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27'}, enabled=[], networks=['172.24.5.4/24'], peer=[], ipv6_ra_configs={}) old= {{(pid=105470) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} Feb 16 19:40:22.715543 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched CREATE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(name=lrp-a3b35ad3-6c89-4653-b097-0f605244fd26, ipv6_prefix=[], mac=fa:16:3e:b5:cd:f1, gateway_chassis=[], external_ids={'neutron:is_ext_gw': 'True', 'neutron:network_name': 'neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071', 'neutron:revision_number': '8', 'neutron:router_name': 'neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4', 'neutron:subnet_ids': '0ee58c77-b622-4ab4-adb2-400b1ab81596 8d726240-563e-4fd4-b3c3-c3b0599bca6a'}, options={'gateway_mtu': '1372', 'reside-on-redirect-chassis': 'true'}, ha_chassis_group=[], status={}, enabled=[], networks=['172.24.5.4/24', '2001:db8::1/64'], peer=[], ipv6_ra_configs={}) old= {{(pid=105470) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} Feb 16 19:40:22.770493 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.common.ovn.utils [-] HA Chassis Group neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994 synchronized; highest priority chassis 4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27 Feb 16 19:40:22.771252 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=HA_Chassis_Group, record=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, col_values=(('external_ids', {'neutron:availability_zone_hints': '', 'neutron:network_id': '1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'neutron:router_id': 'b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4'}),), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:22.771747 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): HAChassisGroupAddChassisCommand(_result=None, hcg=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, chassis=4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27, priority=1, columns={}) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:22.772358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.773538 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Traceback (most recent call last): Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/event.py", line 185, in notify_loop Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event match.run(event, row, updates) Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/services/ovn_l3/ovsdb_monitor.py", line 80, in run Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.l3_plugin._ovn_client.update_router_ha_chassis_group( Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1635, in update_router_ha_chassis_group Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.link_network_ha_chassis_group(context, network_id, router_id) Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2381, in link_network_ha_chassis_group Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with super().transaction(*args, **kwargs) as t: Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:22.774442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self.create_transaction( Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.result = self.commit() Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^ Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise result.ex Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event txn.results.put(txn.do_commit()) Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^^^ Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise RuntimeError(msg) Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.775736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Feb 16 19:40:22.805759 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.common.ovn.utils [-] HA Chassis Group neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994 synchronized; highest priority chassis 4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27 Feb 16 19:40:22.806375 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=HA_Chassis_Group, record=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, col_values=(('external_ids', {'neutron:availability_zone_hints': '', 'neutron:network_id': '1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'neutron:router_id': 'b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4'}),), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:22.806845 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): HAChassisGroupAddChassisCommand(_result=None, hcg=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, chassis=4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27, priority=1, columns={}) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:22.807131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.807888 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Traceback (most recent call last): Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/event.py", line 185, in notify_loop Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event match.run(event, row, updates) Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/services/ovn_l3/ovsdb_monitor.py", line 88, in run Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.l3_plugin._ovn_client.update_router_ha_chassis_group( Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1635, in update_router_ha_chassis_group Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.link_network_ha_chassis_group(context, network_id, router_id) Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2381, in link_network_ha_chassis_group Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with super().transaction(*args, **kwargs) as t: Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:22.808879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self.create_transaction( Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.result = self.commit() Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^ Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise result.ex Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event txn.results.put(txn.do_commit()) Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^^^ Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise RuntimeError(msg) Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:22.809403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Feb 16 19:40:25.633000 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:25.635378 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:25.668854 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:25.669288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.670525 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.671814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.672674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:25.675823 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:25.677529 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:25.677844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.678643 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:25.679943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.680633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:25.724391 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:25.725745 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-71215643-ba0d-4bfa-b9ed-5d7e4ab2cea3 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:25.726681 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:25.727152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-71215643-ba0d-4bfa-b9ed-5d7e4ab2cea3 None None] Traceback (most recent call last): Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.728301 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-71215643-ba0d-4bfa-b9ed-5d7e4ab2cea3 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:25.729842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.730271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-71215643-ba0d-4bfa-b9ed-5d7e4ab2cea3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:25.731679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:25.732157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:25.732634 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:25.732634 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:25.732634 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:25.999335 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:26.000551 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:26.001007 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:26.001265 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:26.001385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:26.001934 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:26.002805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:26.003674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:30.634493 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:30.637727 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:30.667107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:30.667491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.668493 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.669342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.670344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:30.677874 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:30.680544 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:30.680773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.681437 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.683336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.684031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:30.725499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:30.726869 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:30.727555 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:30.727804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.728480 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:30.729959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.730326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:30.731530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:30.731962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:30.732380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:30.732380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:30.732380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:31.000583 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:31.001536 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:31.002061 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:31.002403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:31.002589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:31.003391 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:31.005167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:31.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:35.636023 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:35.641018 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:35.670430 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:35.670828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.671640 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.672667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.673540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:35.680202 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:35.681155 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:35.681337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.681913 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:35.683403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.683926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:35.726723 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:35.727732 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:35.728274 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:35.728578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.729394 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:35.730585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.731381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:35.732072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:35.732827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:35.733623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:35.733623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:35.733623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:36.001767 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:36.002527 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:36.002944 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:36.003213 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:36.003358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:36.004000 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:36.005189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:36.006110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:40.640102 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:40.642617 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:40.681805 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:40.684660 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:40.684904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.688976 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.691875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.692981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.694682 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.695555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.696983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:40.727837 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:40.728980 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:40.729397 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:40.729652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.730366 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:40.731995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.733373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:40.733954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:40.735009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:40.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:40.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:40.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:41.002903 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:41.004127 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:41.004675 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:41.005088 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:41.005276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:41.006117 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:41.007898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:41.008784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:45.642088 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:45.645162 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:45.685810 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:45.692856 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:45.692856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.696050 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:45.700220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.700958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:45.702588 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:45.703108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.704330 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.706283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:45.729387 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:45.731077 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:45.733747 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:45.733747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.734601 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:45.736360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.738078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:45.739100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:45.742451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:46.006744 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:46.006935 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:46.007156 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:46.007415 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:46.007558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:46.008123 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:46.009122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:46.009982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:50.644137 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:50.645682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:50.680006 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:50.680443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.681352 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.682284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.683149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:50.689561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:50.690958 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:50.691299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.692085 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.693655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.694440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:50.730934 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:50.732215 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:50.732644 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:50.733003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:50.737745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.738880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:50.740738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:50.741308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:50.741864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:50.741864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:50.741864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:51.006924 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:51.008251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:51.008786 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:51.009216 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:51.009416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:51.010220 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:51.011973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:51.012814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:55.645593 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:55.647464 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:55.689408 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:55.689848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.690746 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.691158 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.692213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.693172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:55.693995 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:55.693995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.694471 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.695760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.696284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:55.732208 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:55.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:40:55.733653 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:55.733992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.734778 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:40:55.736238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.737042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:40:55.737945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:55.738661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:55.739477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:55.739477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:55.739477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:40:56.010318 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:40:56.011558 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:40:56.012222 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:56.012710 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:40:56.012934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:56.013716 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:56.014854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:40:56.015759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:00.646842 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:00.649146 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:00.685356 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:00.685805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Traceback (most recent call last): Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.686869 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.688063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.688871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:00.692894 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:00.693994 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:00.694321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.695254 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.696370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.696912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:00.733226 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:00.734210 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:00.734724 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:00.735049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.735880 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:00.737219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.738283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:00.738984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:00.740034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:00.741208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:00.741208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:00.741208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:01.012658 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:01.014375 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:01.016441 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:01.017085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:01.017305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:01.018185 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:01.019857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:01.020780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:05.648482 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:05.650898 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:05.689735 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:05.690028 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Traceback (most recent call last): Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.691271 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.692619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.693493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:05.694445 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:05.695278 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:05.695564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.696462 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:05.698674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.699466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:05.734079 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:05.734833 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:05.735403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:05.735663 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.736506 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:05.738037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.739153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:05.740273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:05.742124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:05.743445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:05.743445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:05.743445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:06.013878 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:06.014901 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:06.015601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:06.016307 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:06.016503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:06.017662 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:06.019383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:06.020272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:10.649997 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:10.652329 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:10.654210 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=564,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:41:10.690864 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:10.691132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Traceback (most recent call last): Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.692035 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.693081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.694037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:10.695176 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:10.696415 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:10.696715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.697566 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.700797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:10.735063 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:10.736650 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:10.737252 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:10.737516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.738432 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:10.739437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.740155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:10.740575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:10.741357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:10.742175 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:10.742175 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:10.742175 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:11.015260 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:11.016539 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:11.017355 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:11.017850 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:11.018057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:11.018833 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:11.019994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:11.020822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Traceback (most recent call last): Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.700662 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.702122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.702992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.703849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.704496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.705435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:15.736724 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:15.737797 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:15.738414 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:15.738635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.739722 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:15.741252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.741987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:15.743053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:15.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:15.744664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:15.744664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:15.744664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:16.016599 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:16.017847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:16.018543 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:16.018996 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:16.019186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:16.019958 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:16.021099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:16.021920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:20.652956 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:20.655408 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:20.688670 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:20.688670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Traceback (most recent call last): Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.691174 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.692567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.693511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:20.696844 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:20.697877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:20.698098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.698653 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.699786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.700096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:20.737920 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:20.739016 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:20.739579 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:20.739881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.740588 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:20.741907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.742621 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:20.743532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:20.744569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:20.745596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:20.745596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:20.745596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:21.017947 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:21.019000 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:21.019785 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:21.020274 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:21.020527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:21.021523 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:21.023846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:21.024378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:25.654773 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:25.657303 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:25.687746 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:25.688101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Traceback (most recent call last): Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.689195 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.689877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.690422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:25.697824 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:25.699186 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:25.699461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.700677 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.701685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.702943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:25.739114 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:25.740579 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:25.741408 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:25.741894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.742891 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:25.744714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.745539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:25.747558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:25.748228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:25.748719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:25.748719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:25.748719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:26.019324 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:26.021036 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:26.021509 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:26.021937 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:26.022124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:26.022910 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:26.024760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:26.025391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:30.656503 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:30.658660 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:30.692904 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:30.693300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Traceback (most recent call last): Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.694213 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.695205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.696193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:30.698968 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:30.700544 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:30.700866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.701892 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.703994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.704550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:30.740260 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:30.741398 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:30.742174 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:30.742467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.743292 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:30.744204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.744929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:30.745375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:30.746411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:30.747391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:30.747391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:30.747391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:31.020880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:31.021858 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:31.022576 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:31.023011 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:31.023213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:31.024150 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:31.025925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:31.026626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:35.657882 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:35.660093 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:35.696380 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:35.696603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Traceback (most recent call last): Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.697348 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.698984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.699551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:35.701139 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:35.701556 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:35.701932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.702642 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.705049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.706522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:35.741600 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:35.743052 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:35.743589 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:35.743986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.744858 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:35.746090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.746635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:35.747645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:35.748165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:35.748961 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:35.748961 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:35.748961 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:36.021793 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:36.023028 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:36.023907 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:36.024340 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:36.024540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:36.025537 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:36.027444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:36.028091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:40.659813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:40.661527 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:40.695357 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:40.695568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Traceback (most recent call last): Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.696210 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.697327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.698291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:40.701353 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:40.702354 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:40.702714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.703567 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.704324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.705123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:40.743178 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:40.743855 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:40.745044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:40.745454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.746338 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:40.747714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.748430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:40.749308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:40.750083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:40.750967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:40.750967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:40.750967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:41.023175 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:41.023972 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:41.024893 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:41.025386 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:41.025598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:41.026565 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:41.028397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:41.029155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:45.660494 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:45.662794 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:45.695113 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:45.695372 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.696160 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.697310 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.698268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:45.703924 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:45.705231 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:45.705780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.707141 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.709465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.710014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:45.743932 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:45.744617 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:45.745467 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:45.746353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.747500 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:45.749021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.749874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:45.750728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:45.751673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:45.752567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:45.752567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:45.752567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:46.024195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:46.024999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:46.025812 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:46.026236 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:46.026411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:46.027177 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:46.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:46.029728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:50.661334 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:50.663125 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-e5461a36-53b0-4300-b4a3-ae9d18726acb None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:50.698200 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:50.698617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-e5461a36-53b0-4300-b4a3-ae9d18726acb None None] Traceback (most recent call last): Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.699624 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-e5461a36-53b0-4300-b4a3-ae9d18726acb None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.700894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.702001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:50.704595 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:50.705722 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:50.706073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.706965 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.709209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.710017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:50.744642 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:50.745350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:50.745991 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:50.746328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.747268 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:50.748847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.749580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:50.750512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:50.751389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:50.752262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:50.752262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:50.752262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:51.025054 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:51.025552 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:51.026290 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:51.026684 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:51.026914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:51.027968 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:51.029650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:51.030477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:55.662107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:55.663680 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:55.700572 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:55.701214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.702568 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.705669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.706584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:55.707484 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:55.707484 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:55.707484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.708137 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.710181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.710760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:55.745383 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:55.746091 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:41:55.746847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:55.747176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.748109 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:41:55.748983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.749855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:41:55.750303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:55.751042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:55.751860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:55.751860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:55.751860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:41:56.025813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:41:56.026621 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:41:56.027074 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:56.027505 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:41:56.027773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:56.028622 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:56.030349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:41:56.031291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:00.662992 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:00.665111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:00.695888 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:00.696338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.697518 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.699036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.699987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:00.706526 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:00.707504 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:00.707720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.708558 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:00.710857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.711618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:00.746118 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:00.746472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:00.747107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:00.747420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.748397 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:00.750145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.751109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:00.752214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:00.753080 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:00.754001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:00.754001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:00.754001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:01.026494 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:01.027095 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:01.027571 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:01.027971 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:01.028155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:01.028916 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:01.030499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:01.031369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:05.664020 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:05.666407 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:05.697669 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:05.698215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Traceback (most recent call last): Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.699994 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.700897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.701989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:05.707606 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:05.708227 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:05.708561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.709533 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.711644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.712210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:05.746539 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:05.747057 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:05.747641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:05.747992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.748838 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:05.750955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.751788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:05.752185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:05.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:05.753843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:05.753843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:05.753843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:06.027173 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:06.027846 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:06.028360 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:06.028796 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:06.029036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:06.029837 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:06.031735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:06.032661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:10.665038 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:10.667436 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:10.688051 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=253 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:42:10.696573 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:10.697052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.697996 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.699183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.700172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:10.707810 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:10.709105 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:10.709444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.710250 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.711382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.712273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:10.746981 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:10.747493 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:10.748017 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:10.748353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.749223 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:10.750450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.751200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:10.751930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:10.752686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:10.753532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:10.753532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:10.753532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:11.027844 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:11.028487 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:11.029086 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:11.029366 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:11.029492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:11.030141 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:11.031732 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:11.032530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:15.666246 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:15.668156 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:15.705364 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:15.706098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.707618 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.709199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.710186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:15.710970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:15.710970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:15.710970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.711232 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:15.711974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.712432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:15.747666 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:15.748239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:15.748939 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:15.749511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.750627 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:15.751554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.752550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:15.752978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:15.753800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:15.754674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:15.754674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:15.754674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:16.028554 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:16.029264 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:16.029933 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:16.030371 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:16.030584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:16.031422 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:16.033043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:16.033797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:20.667014 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:20.669182 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:20.714786 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:20.719232 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:20.719583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.720336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:20.720584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Traceback (most recent call last): Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.721358 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.721857 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.723659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.724631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.725557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.726406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:20.748261 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:20.748935 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:20.749408 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:20.749611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.750278 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:20.751722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.752582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:20.753274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:20.754075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:20.754987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:20.754987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:20.754987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:21.029358 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:21.030221 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:21.031082 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:21.031490 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:21.031781 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:21.032689 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:21.034669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:21.035485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:25.668067 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:25.670023 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-42e153b6-b31e-49d5-b232-4a777d2bd4ae None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:25.742741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:25.742741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:25.743263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-42e153b6-b31e-49d5-b232-4a777d2bd4ae None None] Traceback (most recent call last): Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.747118 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-42e153b6-b31e-49d5-b232-4a777d2bd4ae None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.748800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.749277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:25.749796 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:25.750093 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:25.750493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.750864 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:25.751540 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.752126 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:25.752720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.754834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:25.755325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:25.755850 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:25.755850 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:25.755850 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:25.755850 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:25.755850 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:25.755850 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.755850 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:25.756344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.756778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:25.758033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:25.758418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:25.758960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:25.758960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:25.758960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:26.030026 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:26.030717 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:26.031323 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:26.031816 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:26.032047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:26.033041 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:26.034618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:26.035347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:30.668670 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:30.670312 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:30.704322 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:30.704786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.705736 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.707035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.707849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:30.717807 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:30.718607 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:30.718922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.719981 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:30.721683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.722597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:30.750606 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:30.750882 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:30.751195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:30.751387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.751895 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:30.753121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.754324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:30.754774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:30.755533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:30.756427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:30.756427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:30.756427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:31.030829 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:31.031267 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:31.031689 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:31.032167 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:31.032268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:31.033144 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:31.034092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:31.034978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:35.669252 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:35.671084 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:35.699903 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:35.700217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.700918 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.701856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.702676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:35.718380 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:35.719495 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:35.719830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.720845 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.722883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.723624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:35.751168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:35.751791 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:35.752136 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:35.752386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.753793 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:35.754335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.755105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:35.755848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:35.756546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:35.757347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:35.757347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:35.757347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:36.031457 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:36.032269 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:36.032874 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:36.033308 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:36.033631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:36.034849 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:36.036277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:36.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:40.670049 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:40.671373 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:40.706573 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:40.706802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.707393 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.708254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.708746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:40.719026 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:40.719770 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:40.720086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.720868 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.722555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.723320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:40.752028 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:40.752474 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:40.753184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:40.753475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.754274 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:40.755164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.755871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:40.756182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:40.756987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:40.757863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:40.757863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:40.757863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:41.032214 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:41.032675 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:41.033288 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:41.033671 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:41.033911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:41.034670 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:41.035938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:41.036400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:45.671045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:45.672661 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:45.707739 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:45.708166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.709092 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.709929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.710882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:45.719568 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:45.720714 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:45.721056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.722041 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.723969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.724635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:45.752659 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:45.753142 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:45.753602 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:45.753922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.754594 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:45.755826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.756552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:45.757346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:45.758089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:45.758924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:45.758924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:45.758924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:46.032861 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:46.033186 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:46.033824 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:46.034264 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:46.034523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:46.035612 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:46.036879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:46.037268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:50.671938 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:50.673625 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:50.706297 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:50.706461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.707266 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.709157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.709965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:50.720160 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:50.721073 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:50.721382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.722185 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.723982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.724918 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:50.753297 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:50.753846 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:50.754341 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:50.754598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.755317 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:50.756206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.757012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:50.757383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:50.758129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:50.759071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:50.759071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:50.759071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:51.033554 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:51.034204 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:51.035637 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:51.035637 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:51.035637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:51.036593 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:51.038450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:51.039570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:55.672664 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:55.674169 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:55.701798 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:55.702207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.703356 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.704348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.704737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:55.720835 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:55.721793 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:55.722073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.722824 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.724466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.725260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:55.754049 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:55.754383 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:42:55.754951 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:55.755282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.756100 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:42:55.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.757881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:42:55.758267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:55.758963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:55.759741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:55.759741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:55.759741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:42:56.034324 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:42:56.034659 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:42:56.035423 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:56.035866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:42:56.036055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:56.036986 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:56.039471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:42:56.040400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:00.673502 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:00.675337 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:00.707510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:00.707965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.708794 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.709935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.710734 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:00.721360 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:00.722476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:00.722788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Traceback (most recent call last): Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.723587 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:00.724897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.725387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:00.754626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:00.755103 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:00.755476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:00.755682 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.756241 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:00.757013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.757894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:00.758241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:00.758946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:00.759722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:00.759722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:00.759722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:01.035341 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:01.035926 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:01.036585 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:01.037066 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:01.037270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:01.038234 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:01.040595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:01.041472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:05.674130 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:05.675805 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:05.709632 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:05.710105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:05.712006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:05.713196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:05.714018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:05.714018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:05.714018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:05.714018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:05.714018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.714018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:05.722198 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:05.723247 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:05.723556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Traceback (most recent call last): Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.724502 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-81937310-86f7-4a23-8ce2-a00e2333098f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:05.726943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.728137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:05.755376 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:05.756097 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-42e153b6-b31e-49d5-b232-4a777d2bd4ae None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:05.757837 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:05.757837 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-42e153b6-b31e-49d5-b232-4a777d2bd4ae None None] Traceback (most recent call last): Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.758062 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-42e153b6-b31e-49d5-b232-4a777d2bd4ae None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:05.759651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.760345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-42e153b6-b31e-49d5-b232-4a777d2bd4ae None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:05.761463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:05.762938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:05.765046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:05.765046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:05.765046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:06.035909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:06.036345 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:06.037123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:06.037510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:06.037683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:06.038512 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:06.040337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:06.040870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:10.674951 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:10.677228 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:10.696152 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=234 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:43:10.714161 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:10.714475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.715178 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.716379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.717461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:10.722845 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:10.723653 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:10.724012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.724742 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.726543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.727223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:10.755906 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:10.756410 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:10.757236 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:10.757510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Traceback (most recent call last): Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.758364 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:10.759670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.760347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:10.761651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:10.762491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:10.763452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:10.763452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:10.763452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:11.044396 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:11.045062 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:11.045725 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:11.046268 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:11.046515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:11.047678 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:11.049499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:11.050269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:15.676189 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:15.677601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:15.715140 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:15.715583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.716591 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.717600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.718610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:15.726769 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:15.727665 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:15.727886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.728420 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:15.729232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.729540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:15.756487 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:15.756989 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:15.757524 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:15.757939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.758905 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:15.760117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.760884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:15.761715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:15.762438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:15.763233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:15.763233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:15.763233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:16.045222 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:16.045847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:16.046404 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:16.046907 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:16.047131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:16.048064 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:16.050144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:16.050615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:20.676989 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:20.678374 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:20.714111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:20.714584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.715666 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.717137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.717946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:20.727364 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:20.728350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:20.728551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.729336 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:20.731171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.732319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:20.757126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:20.757631 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:20.758263 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:20.758568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.759486 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:20.760779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.761460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:20.762320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:20.763040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:20.764314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:20.764314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:20.764314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:21.045964 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:21.046719 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:21.047535 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:21.048005 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:21.048206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:21.049215 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:21.051487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:21.052359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:25.677823 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:25.679411 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:25.711016 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:25.711499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.712413 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.713688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.714499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:25.728060 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:25.729092 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:25.729449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.730752 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:25.732528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.733296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:25.757648 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:25.758120 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:25.758796 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:25.759177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.760411 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:25.762333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.763102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:25.763946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:25.765155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:25.766105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:25.766105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:25.766105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:26.046620 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:26.047293 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:26.047852 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:26.048264 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:26.048459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:26.049287 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:26.050355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:26.051274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:30.678517 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:30.679933 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:30.711856 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:30.712255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.713251 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.714540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.715137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:30.728635 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:30.729682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:30.729997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.731082 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:30.732339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:30.758232 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:30.758720 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:30.759168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:30.759442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.760277 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:30.761443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.762203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:30.763008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:30.763842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:30.764778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:30.764778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:30.764778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:31.047496 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:31.047726 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:31.048288 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:31.048807 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:31.049053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:31.049911 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:31.050948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:31.051755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:35.679177 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:35.681014 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:35.714390 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:35.714547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.715019 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.716084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.716848 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:35.729203 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:35.730244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:35.730603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.731464 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:35.733145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.734280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:35.759143 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:35.760016 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:35.760718 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:35.761034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.762161 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:35.763665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.764298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:35.765520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:35.765982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:35.766516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:35.766516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:35.766516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:36.048198 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:36.048629 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:36.049186 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:36.049689 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:36.050048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:36.051619 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:36.052570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:36.053430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:40.681598 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:40.683157 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:40.717051 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:40.717270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.717996 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.719014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.719878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:40.729772 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:40.730197 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:40.730313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.730686 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:40.731367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.732263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:40.760055 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:40.760547 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:40.761007 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:40.761272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.762033 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:40.765507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:40.765884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:40.766209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:40.766637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:41.049132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:41.052191 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:41.052871 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:41.053201 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:41.053339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:41.054801 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:41.055609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:41.055978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:45.682368 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:45.683830 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:45.718518 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:45.718755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.719139 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.719952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.720747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:45.730158 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:45.730724 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:45.730875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.731353 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:45.732430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.733128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:45.760729 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:45.761299 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:45.761866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:45.762284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.763128 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:45.764374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.765016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:45.766630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:45.767391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:45.768305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:45.768305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:45.768305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:46.052480 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:46.053111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:46.053572 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:46.054018 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:46.054238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:46.055273 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:46.056809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:46.057834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:50.683088 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:50.684856 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:50.716198 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:50.716515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.717624 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.718614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.718997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:50.730625 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:50.731614 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:50.731959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.732824 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:50.734610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.734920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:50.761359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:50.762050 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:50.763070 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:50.763459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.764550 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:50.766090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.766386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:50.768038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:50.768348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:50.768706 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:50.768706 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:50.768706 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:51.053153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:51.058657 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:51.059664 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:51.060131 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:51.060334 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:51.061217 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:51.062867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:51.063619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:55.683725 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:55.685744 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:55.726665 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:55.727176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.728264 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.729673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.730562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:55.731418 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:55.731890 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:55.732251 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.733241 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:55.734353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.735136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:55.761965 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:55.762574 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:43:55.763125 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:55.763451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.764289 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:43:55.765620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.766437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:43:55.767348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:55.768137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:55.768974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:55.768974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:55.768974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:43:56.059394 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:43:56.059828 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:43:56.060652 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:56.061280 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:43:56.061498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:56.062449 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:56.064533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:43:56.065506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:00.684468 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:00.686311 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:00.730104 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:00.730529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.731603 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.733142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.734005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:00.735024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:00.735024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:00.735024 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.735237 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:00.738148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.738779 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:00.762920 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:00.763499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:00.764086 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:00.764466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.765389 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:00.766647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.767572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:00.768423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:00.769374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:00.770532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:00.770532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:00.770532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:01.059967 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:01.060718 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:01.061571 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:01.062134 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:01.062355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:01.063265 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:01.065012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:01.065857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:05.685820 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:05.687106 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:05.725191 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:05.725191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.725716 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.728040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.728390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:05.733784 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:05.734996 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:05.734996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.735274 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:05.737107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.737450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:05.763943 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:05.764985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:05.765816 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:05.767078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.767346 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:05.769063 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.769895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:05.770882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:05.771726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:05.772334 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:05.772334 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:05.772334 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:06.060811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:06.061500 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:06.062313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:06.062876 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:06.063158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:06.064215 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:06.065943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:06.066671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:10.686494 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:10.687731 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:10.734802 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:10.736064 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:10.736459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.740927 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:10.746615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.747528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:10.752094 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:10.752527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.753450 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.754717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.755555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:10.764655 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:10.765203 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:10.765658 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:10.766007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.766903 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:10.768814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.769836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:10.770567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:10.771393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:10.771955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:10.771955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:10.771955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:11.352184 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:11.354427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:11.356345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:15.687253 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:15.688913 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:15.690970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=253 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:44:15.738113 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:15.739900 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:15.739900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.741359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:15.742084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.742816 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:15.745233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.746094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.747109 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.747465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.748211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:15.765315 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:15.765962 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:15.766498 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:15.766869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.767726 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:15.777084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:15.777526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:15.777998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:15.778567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:15.849865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched CREATE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, ipv6_prefix=[], mac=fa:16:3e:29:33:4b, gateway_chassis=[], external_ids={'neutron:is_ext_gw': 'True', 'neutron:network_name': 'neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071', 'neutron:revision_number': '1', 'neutron:router_name': 'neutron-15d28432-bd1b-492c-b98d-eccecccacd15', 'neutron:subnet_ids': '0ee58c77-b622-4ab4-adb2-400b1ab81596 8d726240-563e-4fd4-b3c3-c3b0599bca6a'}, options={'gateway_mtu': '1430', 'reside-on-redirect-chassis': 'true'}, ha_chassis_group=[], status={}, enabled=[], networks=['172.24.5.78/24', '2001:db8::319/64'], peer=[], ipv6_ra_configs={}) old= {{(pid=105470) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} Feb 16 19:44:16.062207 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:16.062914 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:16.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:16.064760 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:16.065051 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:16.066292 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:16.068187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:16.069109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:18.251099 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched CREATE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(name=lrp-ab9f5960-0de1-4ff9-8ace-5112027b5a84, ipv6_prefix=[], mac=fa:16:3e:c2:8a:ff, gateway_chassis=[], external_ids={'neutron:is_ext_gw': 'False', 'neutron:network_name': 'neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1', 'neutron:revision_number': '1', 'neutron:router_name': 'neutron-15d28432-bd1b-492c-b98d-eccecccacd15', 'neutron:subnet_ids': 'f9fb342f-813a-4388-a760-afa136d8ba8f'}, options={}, ha_chassis_group=[], status={}, enabled=[], networks=['192.168.24.1/24'], peer=[], ipv6_ra_configs={}) old= {{(pid=105470) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} Feb 16 19:44:18.286560 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.common.ovn.utils [-] HA Chassis Group neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1 synchronized; highest priority chassis 4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27 Feb 16 19:44:18.287317 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): HAChassisGroupAddCommand(_result=None, name=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, may_exist=True, columns={'external_ids': {'neutron:availability_zone_hints': '', 'neutron:network_id': '5aec5241-dd49-4846-b0bd-f32c1db95ac1', 'neutron:router_id': '15d28432-bd1b-492c-b98d-eccecccacd15'}}) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:18.287638 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): HAChassisGroupAddChassisCommand(_result=None, hcg=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, chassis=4ef07f5f-c6fe-4bf1-abdf-ff43d461ab27, priority=1, columns={}) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:18.288930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:18.289546 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Traceback (most recent call last): Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/event.py", line 185, in notify_loop Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event match.run(event, row, updates) Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/services/ovn_l3/ovsdb_monitor.py", line 85, in run Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.l3_plugin._ovn_client.link_network_ha_chassis_group( Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2381, in link_network_ha_chassis_group Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with super().transaction(*args, **kwargs) as t: Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event next(self.gen) Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event with self.create_transaction( Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event self.result = self.commit() Feb 16 19:44:18.290655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^ Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise result.ex Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event txn.results.put(txn.do_commit()) Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event ^^^^^^^^^^^^^^^ Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event raise RuntimeError(msg) Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:18.291990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.event Feb 16 19:44:20.687846 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:20.689040 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:20.725100 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:20.725566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.726610 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.727530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.728257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:20.737203 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:20.738509 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:20.738967 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:20.739338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.740284 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:20.742330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.743180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:20.767077 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:20.767773 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:20.768369 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:20.768667 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.769518 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:20.770902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.771610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:20.772868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:20.773639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:20.774550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:20.774550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:20.774550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:21.063213 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:21.064058 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:21.064910 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:21.065302 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:21.065538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:21.066662 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:21.068825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:21.069678 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:25.688803 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:25.690874 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:25.724380 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:25.724580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.725169 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.726077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.726828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:25.738276 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:25.739750 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:25.740183 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:25.740464 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.741361 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:25.743021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.743832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:25.768240 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:25.769272 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:25.769623 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:25.769867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.771825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.772168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:25.772408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:25.772702 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:25.773020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:25.773020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:25.773020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:26.064010 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:26.064554 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:26.065253 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:26.065662 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:26.065929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:26.066807 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:26.068842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:26.069303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:30.689741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:30.691682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:30.896602 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:30.896769 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:30.898164 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:30.899084 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:30.905195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:30.905531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.908044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:30.908277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Traceback (most recent call last): Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.909045 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:30.913008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.913756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.914552 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-22f5ba15-06ef-4d09-9951-9d4da7edfbef None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:30.916221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:30.917094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:30.917966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:30.917966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.917966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.919389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.920173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:30.937204 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:30.937668 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Traceback (most recent call last): Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.938832 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6f5d0366-3f8e-45a5-bdad-bdca6ffba29f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.940020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:30.940778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:31.065048 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:31.065427 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:31.066638 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:31.066841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:31.067028 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:31.067987 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:31.070210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:31.070679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:35.690757 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:35.692950 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:35.730138 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:35.730492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.731547 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.732904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.733444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:35.897459 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:35.898872 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:35.900645 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:35.901392 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:35.901864 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:35.902280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.903613 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.905801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.906768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.907865 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:35.908333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.909103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:35.909522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:35.910405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:35.911401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:35.911401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:35.911401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:36.066570 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:36.066834 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:36.068870 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:36.069219 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:36.069496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:36.070205 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:36.071869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:36.072319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:40.691443 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:40.692525 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:40.727385 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:40.727910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.729000 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.730234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.731045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:40.898553 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:40.901167 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:40.901624 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:40.902905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.903363 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:40.903601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.903877 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.905777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.906824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.907908 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:40.908641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.909430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:40.910313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:40.910830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:40.911456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:40.911456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:40.911456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:41.067239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:41.067620 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:41.068195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:41.068587 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:41.068868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:41.070040 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:41.071719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:41.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:45.693165 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:45.694169 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:45.736947 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:45.737468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.738721 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.740046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.740939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:45.900835 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:45.903728 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:45.904136 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:45.905870 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:45.905870 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:45.906718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.908786 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:45.908786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.909784 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:45.912549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.913094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:45.918644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:45.919125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-71215643-ba0d-4bfa-b9ed-5d7e4ab2cea3 None None] Traceback (most recent call last): Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.919596 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-71215643-ba0d-4bfa-b9ed-5d7e4ab2cea3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:45.920551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:45.920910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:46.067864 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:46.068347 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:46.069239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:46.069637 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:46.069938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:46.070796 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:46.072653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:46.073250 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:50.693161 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:50.714665 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:50.753230 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:50.753230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.753406 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.755025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.756109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:50.901667 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:50.903152 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:50.903458 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:50.903637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.904795 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:50.905293 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.905956 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:50.908170 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.908623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:50.909189 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:50.909189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.910005 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:50.911605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.912481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:50.913325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:50.914189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:50.915096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:50.915096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:50.915096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:51.068807 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:51.069717 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:51.070587 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:51.071062 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:51.071326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:51.072515 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:51.074366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:51.075223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:55.700140 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:55.701667 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:55.737673 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:55.737921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:55.738514 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.163415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.165751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:56.167153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:56.167153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:56.167153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:56.167153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.167153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:56.167153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:44:56.167153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:56.167531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:56.168618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.169645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.170399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.171160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:56.171876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:44:56.172697 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:44:56.173207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:44:56.174014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:44:56.174675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:00.700766 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:00.703098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:00.747684 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:00.748033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.748915 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.750104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.750700 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:00.903086 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:00.905264 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:00.905681 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:00.905977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.907754 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.908246 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:00.908970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.911548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.912086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:00.912587 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:00.912587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.913864 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:00.915509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.916197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:00.917596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:00.918142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:00.918666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:00.918666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:00.918666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:01.070612 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:01.071457 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:01.072382 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:01.072843 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:01.073126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:01.074111 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:01.076101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:01.076913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:05.702178 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:05.703832 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:05.741531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:05.742265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.743338 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.744885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.745505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:05.904422 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:05.907088 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:05.907538 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:05.907807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.909082 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.909882 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:05.912601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.913077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:05.913562 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:05.913727 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:05.914022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.914715 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:05.916955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.917396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:05.919270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:05.919973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:05.920603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:05.920603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:05.920603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:06.071477 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:06.072299 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:06.073164 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:06.073685 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:06.074948 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:06.076996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:06.077727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:10.703064 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:10.705080 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:10.742161 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:10.742453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.743366 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.744826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.745238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:10.906094 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:10.907646 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:10.907975 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:10.908241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Traceback (most recent call last): Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.909348 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:10.911393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.912618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:10.913389 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:10.913389 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-e5461a36-53b0-4300-b4a3-ae9d18726acb None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:10.913389 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:10.913389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-e5461a36-53b0-4300-b4a3-ae9d18726acb None None] Traceback (most recent call last): Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.913573 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-e5461a36-53b0-4300-b4a3-ae9d18726acb None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:10.914717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.915321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-e5461a36-53b0-4300-b4a3-ae9d18726acb None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:10.916254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:10.916766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:10.917255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:10.917255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:10.917255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:11.072148 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:11.073563 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:11.073958 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:11.074293 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:11.074758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:11.075333 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:11.077270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:11.078558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:15.704128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:15.705900 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:15.721325 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=261 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:45:15.743057 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:15.743388 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.744446 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.745530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.746999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:15.907180 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:15.909909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:15.910668 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:15.910956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.912280 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:15.912921 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.914882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.915979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:15.916562 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:15.916562 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:15.916661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Traceback (most recent call last): Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.917566 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:15.918805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.919484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-9ac1c0e5-b2f6-4632-830e-f513ea7fd3d3 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:15.920484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:15.921044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:15.921585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:15.921585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:15.921585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:16.073074 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:16.073726 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:16.074230 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:16.074683 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:16.075004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:16.075876 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:16.077774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:16.078547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:20.638362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:20.650018 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:387}} Feb 16 19:45:20.650278 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Number of inconsistencies found at create/update: networks=4, security_groups=3, routers=1, router_ports=1 {{(pid=105470) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:368}} Feb 16 19:45:20.650406 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:20.699798 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, resource={'id': '1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'name': 'private', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['247b2a4f-27c3-48ff-8b5b-2a6e34d81c2d', '6aa6bb34-b778-4413-99e5-d4ac2fc182da'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:40:06Z', 'updated_at': '2026-02-16T19:40:09Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 59043}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.700113 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.700380 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ddd02391-cc87-498e-9977-c16d9842f554, external_ids_update=None, columns={'tag': 59043}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.700472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.701128 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Failed to fix resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.702180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:45:20.702904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.703561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:45:20.703561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.703561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:45:20.703561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:20.705924 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:20.708790 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:20.758803 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, resource={'id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'name': 'public', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0ee58c77-b622-4ab4-adb2-400b1ab81596', '8d726240-563e-4fd4-b3c3-c3b0599bca6a'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-02-16T19:40:12Z', 'updated_at': '2026-02-16T19:40:21Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.759326 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '0', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.761503 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-baea1945-0145-4a27-8595-4a08d27c2335, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.762006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.763508 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Failed to fix resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.766255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:45:20.766859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.767549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:45:20.767549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.767549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:45:20.767549 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.767549 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:20.768138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.769493 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.771518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.772395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:20.799548 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, resource={'id': '81fd018a-121a-4a03-98c1-081a81a969fa', 'name': 'admin_net', 'tenant_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'project_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['84859701-b3de-4f64-be17-a75ccc95c0e1'], 'standard_attr_id': 42, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:43:33Z', 'updated_at': '2026-02-16T19:43:34Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 13657}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.799815 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, col_values=(('external_ids', {'neutron:network_name': 'admin_net', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.800128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a88b6311-c4bb-44c5-bbf6-b6312d3411d6, external_ids_update=None, columns={'tag': 13657}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.800235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.800892 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Failed to fix resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.801943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:45:20.802493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.803193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:45:20.803193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.803193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:45:20.803193 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:20.830155 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, resource={'id': '5aec5241-dd49-4846-b0bd-f32c1db95ac1', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f9fb342f-813a-4388-a760-afa136d8ba8f'], 'standard_attr_id': 52, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:44:08Z', 'updated_at': '2026-02-16T19:44:10Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 10787}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.830356 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, col_values=(('external_ids', {'neutron:network_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.830715 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb21044d-f495-47d0-9ee3-c77fa26dd090, external_ids_update=None, columns={'tag': 10787}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.830805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.831480 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Failed to fix resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.832553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:45:20.833187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.833817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:45:20.833817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.833817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:45:20.833817 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource 697a2fd9-ba93-42d9-9d78-c7e18fb82403 (type: security_groups) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:20.890997 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.db.ovn_revision_numbers_db [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Successfully bumped revision number for resource 697a2fd9-ba93-42d9-9d78-c7e18fb82403 (type: security_groups) to 9 Feb 16 19:45:20.891240 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource 57570061-ee63-4c60-82b0-1cb1f5fc5514 (type: security_groups) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:20.909620 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:20.911936 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.912382 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.912643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Traceback (most recent call last): Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.914226 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:20.916959 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d9e338dc-bf04-4f2b-a374-1e08f396fe18 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:20.919643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.920419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:20.921295 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:20.921295 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:20.921685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.924502 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:20.925770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.926234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:20.928184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:20.928647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:20.929134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:20.929134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:20.929134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:20.943367 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.db.ovn_revision_numbers_db [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Successfully bumped revision number for resource 57570061-ee63-4c60-82b0-1cb1f5fc5514 (type: security_groups) to 10 Feb 16 19:45:20.943502 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource f3d262f7-9376-4fb3-a9b3-8b6211cbddc3 (type: security_groups) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:20.994534 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.db.ovn_revision_numbers_db [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Successfully bumped revision number for resource f3d262f7-9376-4fb3-a9b3-8b6211cbddc3 (type: security_groups) to 15 Feb 16 19:45:20.994745 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:21.077044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:21.079266 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:21.080636 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.082285 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.082285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.084586 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:21.087046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.088062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:21.108271 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, resource={'id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'enable_snat': True}, 'gw_port_id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'description': 'Created by cluster-api-provider-openstack cluster magnum-system-kube-haue7', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'ip_address': '172.24.5.78', 'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596'}, {'ip_address': '2001:db8::319', 'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-02-16T19:44:11Z', 'updated_at': '2026-02-16T19:44:17Z', 'revision_number': 3, 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf'}, resource_type=routers, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, columns={'external_ids': {'neutron:router_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:21.108515 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:21.108989 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:21.108989 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:21.108989 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.108989 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:21.108989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Unable to update router 15d28432-bd1b-492c-b98d-eccecccacd15. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Failed to fix resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1605, in update_router Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 256, in __exit__ Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Feb 16 19:45:21.113937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 222, in force_reraise Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1539, in update_router Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:45:21.114430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:45:21.114904 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Fixing resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:45:21.243121 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, resource={'id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'name': '', 'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'project_id': '', 'mac_address': 'fa:16:3e:29:33:4b', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'device_owner': 'network:router_gateway', 'standard_attr_id': 57, 'fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000155677', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-02-16T19:44:12Z', 'updated_at': '2026-02-16T19:44:16Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.243421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0ee58c77-b622-4ab4-adb2-400b1ab81596 8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'neutron:network_name': 'neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-15d28432-bd1b-492c-b98d-eccecccacd15'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::319/64', '172.24.5.78/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.244065 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=493a778d-8a03-4344-aca8-f6d87f149d26, lrouter_port=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:21.244452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Traceback (most recent call last): Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.245429 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Failed to fix resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1942, in update_router_port Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:21.246322 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:45:21.246883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:21.247257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:45:21.247257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:21.247257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:45:21.247257 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1b65a8bd-1b21-4705-a3c4-90023876ae41 None None] Maintenance task: Synchronization completed (took 0.60 seconds) Feb 16 19:45:25.708738 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:25.710319 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:25.740026 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:25.740026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.740549 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.741817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.743012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:25.910880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:25.913591 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:25.913591 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:25.913591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Traceback (most recent call last): Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.914271 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:25.916595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.917084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:25.917876 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:25.918376 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:25.919056 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:25.919428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.920535 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:25.922215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.923339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:25.923913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:25.924362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:25.924830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:25.924830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:25.924830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:26.079498 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:26.080009 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:26.082416 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:26.082416 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:26.082416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:26.082624 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:26.084300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:26.085941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:30.709431 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:30.710661 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:30.742375 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:30.743090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.744046 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.744977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.745816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:30.911345 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:30.912589 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:30.913019 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:30.913328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.914373 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.916377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.917127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:30.918319 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:30.918918 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:30.919366 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:30.919613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Traceback (most recent call last): Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.920522 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:30.922228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.922721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-91d65810-046e-4d3f-aa68-9622965203b6 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:30.924004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:30.924539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:30.925030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:30.925030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:30.925030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:31.080109 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:31.080588 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:31.081092 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:31.081372 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:31.081531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:31.082184 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:31.084013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:31.084557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:35.710257 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:35.711834 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:35.748216 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:35.748554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.749376 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.750355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.751259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:35.912261 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:35.913515 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:35.913920 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:35.914178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.915196 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.917306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.918269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:35.919103 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:35.919504 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:35.920096 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:35.920416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.921294 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:35.922220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.922753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:35.923604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:35.924112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:35.924730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:35.924730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:35.924730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:36.081067 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:36.081687 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:36.082474 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:36.082895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:36.083158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:36.084074 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:36.085884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:36.086481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:40.710992 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:40.712680 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:40.747056 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:40.747256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.747856 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.748930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.749684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:40.912705 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:40.913935 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:40.914273 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:40.914586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.915480 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.916747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.917143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:40.919445 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:40.919801 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:40.920348 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:40.920633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.921441 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:40.922850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.923530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:40.924425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:40.924818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:40.925259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:40.925259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:40.925259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:41.081684 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:41.082280 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:41.082949 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:41.083336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:41.083538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:41.084519 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:41.086089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:41.086748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:45.711584 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:45.713311 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:45.752045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:45.752436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.753361 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.754293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.755032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:45.913302 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:45.914686 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:45.915085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:45.915391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.916499 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.918796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.919952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:45.921217 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:45.921217 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:45.921217 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:45.921530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.921968 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:45.923473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.924192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:45.924721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:45.925336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:45.925876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:45.925876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:45.925876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:46.082472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:46.082979 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:46.083540 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:46.083985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:46.084239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:46.085266 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:46.087068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:46.087985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:50.712291 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:50.714116 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:50.752726 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:50.753064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.754023 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.755520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.756364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:50.913958 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:50.915233 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:50.915601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:50.915876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.916883 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.918988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.919745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:50.920561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:50.921087 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:50.921638 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:50.921978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Traceback (most recent call last): Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.922861 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:50.924262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.925025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:50.925901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:50.926519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:50.927151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:50.927151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:50.927151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:51.083306 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:51.083791 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:51.084325 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:51.084823 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:51.085098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:51.086107 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:51.088094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:51.088909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:55.713186 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:55.714659 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:55.753820 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:55.754103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.755001 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.755602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.756556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:55.914772 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:55.915652 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:55.915907 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:55.916048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.916776 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.918510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.919269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:55.920996 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:55.921415 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:45:55.921928 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:55.922483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Traceback (most recent call last): Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.922968 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:45:55.924225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.924766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-4b70c92f-79f5-4108-ace2-9c4a44050cc5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:45:55.925791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:55.926270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:55.926956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:55.926956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:55.926956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:45:56.084107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:45:56.084567 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:45:56.085102 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:56.085552 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:45:56.085929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:56.087161 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:56.089055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:45:56.089856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:00.714195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:00.715907 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:00.749485 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:00.749673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.750409 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.751561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.752863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:00.915182 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:00.916465 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:00.917011 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:00.917302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.918421 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.920082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.920570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:00.921739 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:00.922098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:00.922733 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:00.923087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:00.924600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.925758 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:00.926383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:00.927326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:01.084624 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:01.085181 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:01.086144 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:01.088298 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:01.088402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:01.089120 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:01.091239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:05.714948 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:05.716589 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:05.752159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:05.752578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.753412 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.754332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.755391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:05.915776 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:05.917145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:05.917497 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:05.917778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.919337 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:05.921303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.922101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:05.922959 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:05.922959 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:05.923179 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:05.923280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:05.925133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.925738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:05.926149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:05.926662 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:05.927140 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:05.927140 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:05.927140 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:06.085570 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:06.086175 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:06.086998 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:06.087415 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:06.087669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:06.088571 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:06.090335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:06.091256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:10.716181 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:10.717590 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:10.750377 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:10.750800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.751634 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.752452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.753571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:10.916379 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:10.917925 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:10.918366 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:10.918719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Traceback (most recent call last): Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.920062 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:10.921965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.922484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:10.922995 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:10.922995 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:10.923526 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:10.923845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Traceback (most recent call last): Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.924402 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:10.925330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.925785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5cc882fb-8921-42d9-81cf-92839b917804 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:10.926376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:10.926854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:10.927358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:10.927358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:10.927358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:11.086321 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:11.086817 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:11.087515 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:11.087999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:11.088405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:11.089485 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:11.091526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:11.092376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:15.716658 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:15.718217 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:15.731922 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=490,neutron:UPDATE=3 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:46:15.752999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:15.753150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.753582 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.754387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.755209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:15.917236 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:15.918577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:15.918577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:15.918742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.919223 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.920220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.921196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:15.922985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:15.923572 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:15.923985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:15.924269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Traceback (most recent call last): Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.925563 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:15.926599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.927408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ba83e199-af5a-4edd-89f5-06a20b49ca99 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:15.928597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:15.929493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:15.930179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:15.930179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:15.930179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:16.086940 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:16.087385 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:16.088196 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:16.088530 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:16.088892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:16.089977 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:16.091745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:16.092540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:20.717488 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:20.719866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:20.755127 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:20.755466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.756228 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.757062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.757868 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:20.917992 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:20.919559 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:20.919946 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:20.920199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.921075 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.923004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.923538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:20.923911 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:20.923911 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:20.924138 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:20.924421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.925113 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:20.926282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.926624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:20.927725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:20.928444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:20.929224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:20.929224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:20.929224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:21.087780 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:21.088802 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:21.089752 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:21.090471 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:21.090859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:21.092136 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:21.095073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:21.096285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:25.719101 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:25.720777 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:25.757566 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:25.758018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.758581 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.760210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:25.918791 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:25.920075 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:25.920365 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:25.920578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.921658 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.922711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.923477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:25.924347 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:25.924409 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:25.924910 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:25.925087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.925622 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:25.927039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.927504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:25.928854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:25.929293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:25.929806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:25.929806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:25.929806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:26.088631 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:26.089316 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:26.089913 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:26.090292 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:26.090529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:26.091355 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:26.092985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:26.093862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:30.720001 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:30.721754 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:30.751622 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:30.752059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.752945 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.753835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.754776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:30.919672 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:30.921270 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:30.921625 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:30.921977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.922958 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.924953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.926056 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:30.926877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:30.926877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:30.926877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:30.926877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.927304 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:30.928648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.929483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:30.930178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:30.930959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:30.931760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:30.931760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:30.931760 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:31.089494 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:31.089750 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:31.090372 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:31.090809 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:31.091055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:31.091853 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:31.093141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:31.094160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:35.720903 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:35.722296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:35.763550 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:35.764544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.765280 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.767377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.768800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:35.920595 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:35.921895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:35.922239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:35.922486 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.923452 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.924836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.925650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:35.926414 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:35.926414 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:35.926414 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:35.926619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.927198 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:35.927962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.928783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:35.929481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:35.930289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:36.090109 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:36.090728 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:36.091527 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:36.091972 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:36.092216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:36.093144 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:36.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:36.095782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:40.721397 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:40.723315 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:40.765302 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:40.765624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.765956 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.766917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.767681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:40.921245 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:40.922345 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:40.922578 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:40.922762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.923491 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.924710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.925468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:40.926191 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:40.926713 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:40.927165 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:40.927165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.927639 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:40.929094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.930531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:40.931231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:40.931937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:40.933228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:40.933228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:40.933228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:41.090621 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:41.091324 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:41.091977 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:41.092445 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:41.092668 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:41.093527 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:41.095115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:41.095980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:45.722035 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:45.723302 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:45.759123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:45.759294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.759754 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.760555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.761604 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:45.922048 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:45.923605 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:45.924020 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:45.924303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.925457 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.927180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.928489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:45.929359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:45.929359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:45.929359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:45.929359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.930361 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:45.932152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.932661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:45.934277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:45.934823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:45.935358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:45.935358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:45.935358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:46.091594 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:46.092079 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:46.092808 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:46.093357 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:46.093713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:46.094809 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:46.096559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:46.097378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:50.722576 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:50.724121 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:50.758276 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:50.758466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.759070 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.759906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.760958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:50.923138 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:50.924805 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:50.925204 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:50.925451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.926429 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.928495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.929263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:50.930309 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:50.930309 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:50.930309 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:50.930675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.931104 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:50.932645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.933194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:50.934346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:50.934818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:50.935328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:50.935328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:50.935328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:51.092007 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:51.092464 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:51.092922 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:51.093202 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:51.093372 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:51.093992 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:51.095052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:51.095818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:55.723178 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:55.724732 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:55.768244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:55.768684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.769819 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.770898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.772867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:55.924052 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:55.925952 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:55.926438 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:55.926712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.927935 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.929975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.930818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:55.931745 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:55.931745 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:46:55.931745 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:55.931946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.932686 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:46:55.934115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.934794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:46:55.936090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:55.936899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:55.937463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:55.937463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:55.937463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:46:56.094126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:46:56.094126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:46:56.094126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:56.094126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:46:56.094126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:56.094774 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:56.095664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:46:56.096352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:00.723785 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:00.725431 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:00.758626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:00.758875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.759549 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.760409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.761407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:00.924748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:00.927200 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:00.927612 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:00.927910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.928971 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.930876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.931400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:00.931897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:00.931897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:00.933595 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:00.933977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.934671 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:00.935287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.935864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:00.936074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:00.936648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:00.937236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:00.937236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:00.937236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:01.093295 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:01.094002 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:01.097535 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:01.097830 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:01.097979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:01.098820 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:01.100797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:01.101291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:05.724621 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:05.726299 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:05.763613 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:05.764122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.765255 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.766381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.767174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:05.925350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:05.926591 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:05.926964 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:05.927204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.928244 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.930190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.931040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:05.932235 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:05.932614 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:05.933209 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:05.933501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.934246 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:05.935600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.936117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:05.937258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:05.937726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:05.938332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:05.938332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:05.938332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:06.094100 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:06.094624 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:06.095241 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:06.095584 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:06.095913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:06.096713 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:06.097906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:06.098947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:10.725344 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:10.727218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:10.769005 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:10.769372 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.770343 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.771618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.772438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:10.925954 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:10.927676 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:10.928025 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:10.928224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.929030 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.931102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.931789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:10.932688 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:10.933196 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:10.933718 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:10.933939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.934520 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:10.935920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.936453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:10.937423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:10.937932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:10.938605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:10.938605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:10.938605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:11.094742 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:11.095124 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:11.095626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:11.095944 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:11.096106 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:11.096682 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:11.097596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:11.098400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:15.726088 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:15.728096 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:15.750364 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=247 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:47:15.765079 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:15.765384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.766165 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.767227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.768109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:15.926740 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:15.928101 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:15.928329 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:15.928481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.929390 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.931336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.931924 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:15.933189 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:15.933705 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:15.934272 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:15.934571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.935301 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:15.936710 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.937271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:15.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:15.938840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:15.939418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:15.939418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:15.939418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:16.095401 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:16.095772 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:16.096729 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:16.097202 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:16.097467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:16.098513 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:16.099776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:16.100659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:20.726831 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:20.728336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:20.765519 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:20.766033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.767102 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.768130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.768885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:20.927381 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:20.928511 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:20.928881 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:20.929270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.930483 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.932361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.932916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:20.933633 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:20.934330 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:20.935035 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:20.935387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.936253 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:20.938810 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.939538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:20.940116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:20.940581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:20.941132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:20.941132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:20.941132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:21.096132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:21.096527 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:21.097340 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:21.097781 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:21.098086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:21.099084 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:21.100774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:21.101330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:25.727543 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:25.729123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:25.766477 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:25.767035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.768122 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.769091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:25.928215 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:25.930019 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:25.930496 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:25.930819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.932192 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.933480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.934443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:25.935362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:25.935362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:25.935575 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:25.935778 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.936373 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:25.937851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.938333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:25.939540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:25.940112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:25.940642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:25.940642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:25.940642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:26.096660 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:26.096916 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:26.097560 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:26.097921 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:26.098122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:26.098798 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:26.099666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:26.100798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:30.728241 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:30.730619 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:30.772223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:30.772682 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.773854 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.775574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.776449 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:30.929141 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:30.930534 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:30.931038 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:30.931308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.932364 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.934098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.935472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:30.936281 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:30.936281 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:30.936368 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:30.936712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.937615 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:30.939078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.939670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:30.940808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:30.941277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:30.941762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:30.941762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:30.941762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:31.097357 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:31.097885 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:31.098646 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:31.099145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:31.099392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:31.100262 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:31.102027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:31.102883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:35.729177 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:35.731220 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:35.769132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:35.769606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.770716 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.772120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.773231 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:35.929775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:35.931197 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:35.931521 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:35.931886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.932839 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.934118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:35.935885 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:35.936235 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:35.936806 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:35.937109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.937886 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:35.939091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.939575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:35.940809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:35.941268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:35.941783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:35.941783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:35.941783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:36.098202 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:36.098953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:36.099496 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:36.099827 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:36.099999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:36.100719 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:36.102461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:36.103283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:40.729882 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:40.731506 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:40.766157 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:40.766552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Traceback (most recent call last): Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.767428 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c8048aed-d36f-4252-aedb-0a6ed2c8abd7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.768716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.769612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:40.930424 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:40.931709 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:40.932090 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:40.932340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.933327 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.934259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.934708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:40.936235 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:40.936610 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:40.937126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:40.937418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.938143 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:40.938866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.939727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:40.940515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:40.941348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:41.098999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:41.099518 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:41.100211 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:41.100707 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:41.101057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:41.101915 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:41.102892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:41.103637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:45.730560 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:45.732134 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:45.762716 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:45.762874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.763314 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.764165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.764603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:45.931285 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:45.932519 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:45.932905 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:45.933212 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.934121 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.935785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.936645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:45.937556 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:45.937556 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:45.937805 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:45.938254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.939124 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:45.940561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.941151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:45.942384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:45.943247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:45.943966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:45.943966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:45.943966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:46.099663 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:46.100890 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:46.101613 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:46.102195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:46.102453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:46.103439 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:46.105515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:46.106329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:50.731256 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:50.733253 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:50.771965 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:50.772233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.773001 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.773973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.774496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:50.931955 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:50.933276 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:50.933597 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:50.933867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.934713 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.936278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.937274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:50.938132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:50.938132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:50.938132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:50.938132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.938390 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:50.939729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.940649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:50.941468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:50.942639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:50.943487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:50.943487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:50.943487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:51.100425 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:51.103331 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:51.104036 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:51.104460 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:51.104716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:51.106999 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:51.108816 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:51.110026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:55.732103 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:55.733185 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:55.770246 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:55.771145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.771917 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.772622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.773633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:55.932620 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:55.934531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:55.935043 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:55.935398 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.936797 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.938371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.938995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:55.939577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:55.939577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:47:55.939739 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:55.939998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.940600 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:47:55.942097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.942832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:47:55.943873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:55.944410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:55.945229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:55.945229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:55.945229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:47:56.103352 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:47:56.104128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:47:56.104720 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:56.105231 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:47:56.105410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:56.106309 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:56.108122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:47:56.109219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:00.732867 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:00.734571 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:00.769395 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:00.769814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.770814 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.772100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.772874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:00.933393 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:00.934650 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:00.935070 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:00.935369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.936388 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.938243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.939396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:00.940528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:00.940528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:00.940528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:00.940528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.941184 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:00.942653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.943414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:00.943835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:00.944501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:00.945156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:00.945156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:00.945156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:01.103999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:01.104339 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:01.104651 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:01.104961 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:01.105103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:01.105687 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:01.106688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:01.107625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:05.733519 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:05.735839 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:05.778263 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:05.778714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.779599 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.780756 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.781279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:05.934193 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:05.935283 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:05.935446 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:05.935627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.936214 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.937245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.938294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:05.939291 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:05.940358 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:05.940947 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:05.941145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.941718 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:05.942893 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.943815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:05.944264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:05.945086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:05.946010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:05.946010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:05.946010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:06.104556 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:06.104942 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:06.105306 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:06.105634 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:06.105878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:06.106484 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:06.107681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:06.108785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:10.734326 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:10.735775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:10.773720 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:10.774239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.775117 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.776099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.776849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:10.934878 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:10.936093 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:10.936476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:10.936708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.937669 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.939617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.940459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:10.941359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:10.941359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:10.941359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:10.941359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.941821 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:10.943221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.944025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:10.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:10.945344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:10.945866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:10.945866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:10.945866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:11.105045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:11.105362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:11.105733 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:11.106024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:11.106196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:11.106794 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:11.107892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:11.108789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:15.735305 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:15.736931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:15.755940 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=239 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:48:15.772128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:15.772550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.773783 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.775709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.776433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:15.935552 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:15.937214 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:15.937773 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:15.937897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.938953 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.940891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.941676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:15.942478 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:15.942478 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:15.942620 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:15.942856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.943351 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:15.944194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.944976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:15.945303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:15.945862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:15.946399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:15.946399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:15.946399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:16.105800 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:16.106268 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:16.106997 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:16.107293 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:16.107446 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:16.108118 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:16.109438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:16.110323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:20.736160 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:20.738098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:20.772507 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:20.772979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.774230 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.775641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.776647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:20.936791 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:20.938504 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:20.939045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:20.939414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.940938 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.942356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.943518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:20.944568 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:20.944568 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:20.944568 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:20.944568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Traceback (most recent call last): Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.945124 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:20.946498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.947452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-b85aaca5-7333-4d84-9f2c-575e83e7de36 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:20.948674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:20.949288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:20.949834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:20.949834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:20.949834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:21.106477 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:21.107257 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:21.107728 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:21.108174 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:21.108427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:21.109743 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:21.111851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:21.112409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:25.737041 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:25.739190 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:25.781342 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:25.781712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.782607 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.783475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.784289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:25.937327 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:25.938728 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:25.939021 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:25.939172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.939844 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.941349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.942219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:25.943363 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:25.944203 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:25.945015 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:25.945328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.946451 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:25.948120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.948680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:25.950208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:25.950780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:25.951421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:25.951421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:25.951421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:26.107305 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:26.108018 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:26.108818 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:26.109322 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:26.109627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:26.110456 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:26.112195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:26.113073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:30.737815 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:30.739903 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:30.778916 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:30.779419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.780595 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.782135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.783195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:30.938182 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:30.939689 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:30.940103 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:30.940353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.941412 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.943496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.944489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:30.945439 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:30.945439 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:30.945439 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:30.945792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.946739 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:30.947737 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.948632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:30.949196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:30.950092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:30.951053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:30.951053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:30.951053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:31.108006 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:31.109231 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:31.110191 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:31.110608 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:31.110899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:31.112256 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:31.114273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:31.115233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:35.738629 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:35.740397 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:35.780238 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:35.780833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.782071 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.783087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.783582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:35.938905 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:35.940642 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:35.941156 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:35.941490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.942730 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.944852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.945211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:35.945595 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:35.946081 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:35.947063 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:35.947601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.948408 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:35.949881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.950278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:35.951553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:35.951947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:35.952461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:35.952461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:35.952461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:36.109126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:36.110194 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:36.110895 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:36.111264 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:36.111539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:36.112490 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:36.115014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:36.116040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:40.739078 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:40.740854 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:40.781476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:40.782356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.783124 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.783879 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.784776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:40.939522 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:40.940807 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:40.941060 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:40.941229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.942061 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.943928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.944617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:40.946072 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:40.946480 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:40.946825 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:40.947036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.953438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.954223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:40.954657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:40.955122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:40.955645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:40.955645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:40.955645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:41.110108 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:41.111585 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:41.112343 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:41.112832 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:41.113014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:41.114125 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:41.116822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:41.117883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:45.739728 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:45.743188 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:45.778985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:45.779552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.780525 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.782127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.782973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:45.940245 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:45.941489 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:45.941931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:45.942271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.943259 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.945277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.946172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:45.946955 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:45.946955 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:45.947099 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.948252 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:45.949321 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:45.950615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:45.951013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:45.951626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:46.110783 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:46.111166 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:46.111681 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:46.111988 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:46.112155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:46.112943 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:46.114108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:46.115351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:50.742170 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:50.745334 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:50.781834 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:50.782081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.782711 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.784042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.784823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:50.940999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:50.946116 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:50.946512 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:50.946711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.948046 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.949549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.950144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:50.951078 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:50.952550 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:50.952550 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:50.952550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.953305 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:50.954825 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.955301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:50.956049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:50.956544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:50.957517 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:50.957517 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:50.957517 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:51.111626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:51.112203 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:51.112803 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:51.113083 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:51.113261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:51.114019 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:51.116546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:51.117947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:55.743886 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:55.746223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:55.789007 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:55.789608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.790832 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.791952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.793377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:55.944326 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:55.946009 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:55.946541 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:55.946933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.948038 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.949185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.950670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:55.951824 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:55.952228 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:48:55.952623 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:55.952963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.953666 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:48:55.954714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.955482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:48:55.956082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:55.956861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:55.957979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:55.957979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:55.957979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:48:56.112511 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:48:56.113542 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:48:56.114901 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:56.115592 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:48:56.116032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:56.117932 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:56.120417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:48:56.121726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:00.744511 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:00.746227 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.777018 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.778408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.779330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:00.945054 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:00.947290 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:00.947290 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:00.947290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.947593 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.949400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.950084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:00.952162 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:00.952825 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:00.953172 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:00.953467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.954191 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:00.955843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.956452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:00.957904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:00.960011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:00.960735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:00.960735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:00.960735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:01.113248 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:01.114232 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:01.116459 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:01.116459 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:01.116459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:01.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:01.119193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:01.121684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:05.745261 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:05.746295 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:05.784296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:05.784649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.785394 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.786399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.787379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:05.945873 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:05.946785 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:05.946973 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:05.947096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.947655 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.948548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.949116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:05.952662 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:05.952948 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:05.953280 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:05.953280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.953637 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:05.954194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.954793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:05.955110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:05.955631 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:05.956326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:05.956326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:05.956326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:06.113960 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:06.114476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:06.115111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:06.115381 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:06.115542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:06.116679 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:06.118436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:06.119381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:10.746097 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:10.748100 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:10.787827 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:10.788204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.789086 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.790566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.791292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:10.946836 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:10.948073 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:10.968497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:10.969568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:10.970611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:10.971273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.972073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:10.972863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:10.973615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:10.974467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:10.974467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:10.974467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:11.115035 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:11.115871 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:11.116667 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:11.117186 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:11.117524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:11.118743 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:11.120982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:11.121923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:15.747619 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:15.749790 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:15.790488 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:15.790853 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.791823 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.792994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.794041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:15.947411 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:15.948748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:15.949200 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:15.949469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.951164 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.952447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.953488 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:15.954334 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:15.954602 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:15.954873 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:15.955068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.955600 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:15.956282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:15.957174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:15.957929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:15.958528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:16.129775 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:16.130266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:16.130587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:16.130977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:16.130977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:20.777230 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:20.777790 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:20.778601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=250 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:49:20.811917 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:20.812292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.812773 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.813681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.814798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:20.948510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:20.949583 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:20.949901 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:20.950118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.950910 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.952367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.953543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:20.954986 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:20.955394 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:20.956075 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:20.956335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.957095 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:20.958808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.959337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:20.960934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:20.961833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:20.962376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:20.962376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:20.962376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:21.117109 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:21.117690 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:21.118197 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:21.118595 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:21.118842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:21.122495 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:21.124133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:21.125029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:25.776932 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:25.778184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:25.818301 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:25.818301 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.818486 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.819509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.820351 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:25.949282 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:25.952070 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:25.952070 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:25.952070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.952325 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.953470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.954046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:25.955299 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:25.955707 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:25.957167 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:25.957167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.957482 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:25.959069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.959557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:25.961609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:25.962338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:25.963004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:25.963004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:25.963004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:26.117887 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:26.120430 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:26.120430 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:26.120430 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:26.120430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:26.121200 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:26.123029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:26.123908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:30.778005 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:30.779199 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:30.811730 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:30.812052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.813004 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.814223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.815004 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:30.949932 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:30.951219 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:30.951543 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:30.951677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.952535 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.954384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.955407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:30.956438 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:30.956438 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:30.956897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:30.957195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.957608 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:30.959092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.959945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:30.960986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:30.961471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:30.961970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:30.961970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:30.961970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:31.118865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:31.119893 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:31.120900 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:31.121332 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:31.121626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:31.122582 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:31.124628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:31.125720 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:35.778811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:35.779791 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:35.824741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:35.826006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.826574 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.829025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.829646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:35.950738 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:35.953491 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:35.953491 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:35.953491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.953875 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.955399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.956078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:35.956630 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:35.956630 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:35.957112 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:35.957565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.958586 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:35.959319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.959847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:35.960121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:35.960689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:35.961210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:35.961210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:35.961210 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:36.119637 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:36.120628 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:36.123046 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:36.123046 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:36.123046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:36.123382 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:36.125992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:36.126911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:40.779408 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:40.781057 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:40.816551 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:40.817348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:40.818418 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:40.819343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:40.820305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.002611 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.003882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.004646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.005530 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:41.006505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:41.007384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:41.007945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.008480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:41.120398 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:41.121190 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:41.158044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:41.159582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:41.160952 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:45.780414 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:45.782667 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:45.820601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:45.820933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.821658 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.822741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.823204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:45.952147 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:45.957209 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:45.958719 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:45.959112 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:45.959344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.961081 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.961751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.962686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:45.964150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:45.965163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:45.966160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:45.967168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:45.968401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:45.968401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:45.968401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:45.968401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:45.968401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:45.968401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:45.968401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:46.138472 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:46.139326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:46.140192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:46.140956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:46.140956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:50.781056 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:50.783241 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:50.814797 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:50.815154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.816187 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.817267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.818474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:50.958274 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:50.960237 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:50.960375 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:50.960524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.961263 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:50.962593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.963152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:50.963746 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:50.963746 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:50.963926 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:50.964121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.964852 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:50.966276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.966785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:50.967903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:50.968459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:50.968977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:50.968977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:50.968977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:51.121605 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:51.122315 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:51.122892 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:51.123095 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:51.123224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:51.123936 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:51.125987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:51.126481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:55.781830 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:55.784384 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:55.817899 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:55.818450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.819616 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.820838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.821253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:55.959715 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:55.961381 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:55.961871 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:55.962203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.963405 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:55.965130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.965983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:55.966488 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:55.966488 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:49:55.966649 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:55.966947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.967592 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:49:55.969297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.969787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:49:55.971047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:55.971483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:55.972323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:55.972323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:55.972323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:49:56.122294 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:49:56.123237 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:49:56.124031 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:56.124350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:49:56.124528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:56.125338 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:56.126684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:49:56.127258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:00.782563 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:00.784056 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:00.819644 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:00.820068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.821316 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.822642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.823249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:00.960272 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:00.962545 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:00.963142 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:00.963673 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.964963 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:00.967219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.968248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:00.969499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:00.969499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:00.969499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:00.969499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.969862 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:00.971786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.973010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:00.973998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:00.974715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:00.975115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:00.975115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:00.975115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:01.123371 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:01.124071 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:01.125090 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:01.125568 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:01.125928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:01.127293 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:01.129476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:01.130332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:05.783353 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:05.784160 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:05.819112 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:05.819359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.820190 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.821447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.822292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:05.961735 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:05.963567 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:05.963567 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:05.964010 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.964884 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:05.966409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.967242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:05.968258 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:05.968909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:05.969405 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:05.970105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.970387 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:05.971891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.972910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:05.973771 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:05.974327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:05.974898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:05.974898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:05.974898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:06.124061 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:06.124531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:06.125259 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:06.125545 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:06.125707 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:06.126420 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:06.128158 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:06.128866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:10.784032 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:10.784032 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:10.831630 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.841858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:10.842975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:10.844044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:10.844044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:10.844044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:10.844044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:10.844044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:10.844044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.844044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:10.962447 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:10.964362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:10.964951 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:10.965294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.966570 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:10.967887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.968787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:10.969685 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:10.969685 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:10.969858 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:10.970039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.970804 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:10.972335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.973199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:10.974042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:10.975194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:10.976166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:10.976166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:10.976166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:11.173953 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:11.175228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:11.176490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:11.177468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:11.177468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:15.784682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:15.786254 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:15.831798 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:15.832366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.833501 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.834846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.835808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:15.963095 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:15.964259 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:15.964520 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:15.964687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.965579 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:15.967412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.967917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:15.968847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:15.969394 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:15.969813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:15.970096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.970838 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:15.972327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.973219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:15.974519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:15.975092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:15.975679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:15.975679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:15.975679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:16.125789 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:16.126537 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:16.127142 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:16.127476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:16.127645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:16.128526 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:16.132683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:16.133568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:20.640274 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:20.648192 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:387}} Feb 16 19:50:20.648373 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Number of inconsistencies found at create/update: networks=4, routers=1, router_ports=1 {{(pid=105470) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:368}} Feb 16 19:50:20.648373 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Fixing resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:50:20.688931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, resource={'id': '1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'name': 'private', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['247b2a4f-27c3-48ff-8b5b-2a6e34d81c2d', '6aa6bb34-b778-4413-99e5-d4ac2fc182da'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:40:06Z', 'updated_at': '2026-02-16T19:40:09Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 59043}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.689437 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.690044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ddd02391-cc87-498e-9977-c16d9842f554, external_ids_update=None, columns={'tag': 59043}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.690402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.691417 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Failed to fix resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.692728 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:50:20.693253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.693841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:50:20.693841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.693841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:50:20.693841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Fixing resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:50:20.720897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, resource={'id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'name': 'public', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0ee58c77-b622-4ab4-adb2-400b1ab81596', '8d726240-563e-4fd4-b3c3-c3b0599bca6a'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-02-16T19:40:12Z', 'updated_at': '2026-02-16T19:40:21Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.721434 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '0', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.722356 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-baea1945-0145-4a27-8595-4a08d27c2335, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.722874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.724069 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Failed to fix resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.725587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:50:20.726166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.726806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:50:20.726806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.726806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:50:20.726806 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Fixing resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:50:20.754665 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, resource={'id': '81fd018a-121a-4a03-98c1-081a81a969fa', 'name': 'admin_net', 'tenant_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'project_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['84859701-b3de-4f64-be17-a75ccc95c0e1'], 'standard_attr_id': 42, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:43:33Z', 'updated_at': '2026-02-16T19:43:34Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 13657}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.755159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, col_values=(('external_ids', {'neutron:network_name': 'admin_net', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.755515 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a88b6311-c4bb-44c5-bbf6-b6312d3411d6, external_ids_update=None, columns={'tag': 13657}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.755738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.756811 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Failed to fix resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.758285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:50:20.758649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.759098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:50:20.759098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.759098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:50:20.759098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Fixing resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:50:20.785618 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:20.790601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:20.792601 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, resource={'id': '5aec5241-dd49-4846-b0bd-f32c1db95ac1', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f9fb342f-813a-4388-a760-afa136d8ba8f'], 'standard_attr_id': 52, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:44:08Z', 'updated_at': '2026-02-16T19:44:10Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 10787}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.793201 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, col_values=(('external_ids', {'neutron:network_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.793859 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb21044d-f495-47d0-9ee3-c77fa26dd090, external_ids_update=None, columns={'tag': 10787}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.794086 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.795510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=325 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.797463 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Failed to fix resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.802578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:50:20.802957 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.803330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:50:20.803330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.803330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:50:20.803330 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Fixing resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:50:20.887649 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.888632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.889475 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.893829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.894382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:20.934104 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, resource={'id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'enable_snat': True}, 'gw_port_id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'description': 'Created by cluster-api-provider-openstack cluster magnum-system-kube-haue7', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'ip_address': '172.24.5.78', 'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596'}, {'ip_address': '2001:db8::319', 'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-02-16T19:44:11Z', 'updated_at': '2026-02-16T19:44:17Z', 'revision_number': 3, 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf'}, resource_type=routers, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.934672 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, columns={'external_ids': {'neutron:router_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.935239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.935732 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.936033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.937068 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.937394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Unable to update router 15d28432-bd1b-492c-b98d-eccecccacd15. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Failed to fix resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1605, in update_router Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 256, in __exit__ Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Feb 16 19:50:20.938429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 222, in force_reraise Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1539, in update_router Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:50:20.938783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:50:20.939149 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Fixing resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:50:20.964057 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:20.970843 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:20.971441 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:20.972110 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.973712 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.973932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.975776 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.977612 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:20.980469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.981290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:20.983172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Traceback (most recent call last): Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.986177 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:20.987283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.987908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-d05ce9ee-f05b-48fc-9e1d-7fdbdd66607c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:20.988970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:20.989763 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:20.990337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:20.990337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:20.990337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:21.091755 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, resource={'id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'name': '', 'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'project_id': '', 'mac_address': 'fa:16:3e:29:33:4b', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'device_owner': 'network:router_gateway', 'standard_attr_id': 57, 'fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000155677', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-02-16T19:44:12Z', 'updated_at': '2026-02-16T19:44:16Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:21.092339 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0ee58c77-b622-4ab4-adb2-400b1ab81596 8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'neutron:network_name': 'neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-15d28432-bd1b-492c-b98d-eccecccacd15'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::319/64', '172.24.5.78/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:21.092830 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=493a778d-8a03-4344-aca8-f6d87f149d26, lrouter_port=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:21.093201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.094953 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Failed to fix resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1942, in update_router_port Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:21.096424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:50:21.096725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:21.097068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:50:21.097068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.097068 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:50:21.097068 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Maintenance task: Synchronization completed (took 0.45 seconds) Feb 16 19:50:21.126484 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:21.127159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:21.128024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:21.128587 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:21.128891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.129925 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:21.131795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:21.132651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:25.787775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:25.789529 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:25.831184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:25.831853 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.832964 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.834152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.835013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:25.965741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:25.967318 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:25.967802 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:25.968047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.969095 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.970620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.971103 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:25.971790 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:25.972007 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:25.972654 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:25.973127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.974145 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:25.975329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.976215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:25.977038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:25.977970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:26.127641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:26.128014 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:26.128963 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:26.129356 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:26.129555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:26.130402 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:26.131835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:26.132272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:30.788468 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:30.789978 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:30.827034 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:30.827034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.828023 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.829174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.830105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:30.966478 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:30.967989 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:30.968293 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:30.968477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.969457 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.971359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.971944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:30.972481 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:30.972481 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:30.973017 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:30.973017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.973513 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:30.974776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.975327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:30.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:30.976916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:30.977421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:30.977421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:30.977421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:31.131945 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:31.132863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:31.133823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:31.134793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:31.134793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:35.789197 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:35.790118 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:35.824274 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:35.824714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.826119 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.827465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.828494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:35.967123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:35.970636 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:35.971200 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:35.971565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.972639 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:35.973311 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.973915 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.976331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.977087 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:35.978260 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:35.978608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.979497 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:35.981073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.981639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:35.982581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:35.983094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:35.983648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:35.983648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:35.983648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:36.128955 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:36.129343 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:36.129991 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:36.130555 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:36.130555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:36.131524 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:36.132946 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:36.133441 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:40.789640 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:40.791935 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:40.830780 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:40.831044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.831527 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.832485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.833514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:40.967886 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:40.968938 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:40.969338 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:40.969676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.970714 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.972638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.973172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:40.973730 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:40.973730 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:40.973997 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:40.974283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.975287 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:40.976624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.977161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:40.978261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:40.978746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:40.979269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:40.979269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:40.979269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:41.129556 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:41.130377 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:41.130841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:41.131145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:41.131323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:41.132033 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:41.133718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:41.134457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:45.791135 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:45.792980 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:45.827657 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:45.827992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.828683 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.829591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.830490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:45.969189 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:45.973597 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:45.975228 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:45.975364 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:45.975477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.975972 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.976950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.977349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:45.977748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:45.977748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:45.977748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.977844 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:45.978479 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.978958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:45.979415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:45.979798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:45.980137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:45.980137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:45.980137 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:46.130422 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:46.131243 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:46.131901 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:46.132349 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:46.132635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:46.133729 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:46.135919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:46.136872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:50.791960 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:50.793679 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:50.839520 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:50.839833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.840805 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.842845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.843450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:50.974343 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:50.976024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:50.976431 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:50.976717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.977796 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:50.979067 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.979495 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:50.980260 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.982806 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.983547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:50.984083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.984511 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:50.985307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.985971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:50.986647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:50.987272 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:50.987914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:50.987914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:50.987914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:51.131239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:51.131857 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:51.132274 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:51.132539 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:51.132738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:51.133329 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:51.134340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:51.135608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:55.792688 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:55.793978 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:55.834671 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:55.835146 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.835892 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.837060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.838047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:55.975065 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:55.976758 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:55.976963 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:55.977305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.978071 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.980711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.981457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:55.982168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:55.982168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:50:55.982784 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:55.983100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.983992 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:50:55.984941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.985412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:50:55.986130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:55.986570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:55.987096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:55.987096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:55.987096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:50:56.132004 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:50:56.132615 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:50:56.133125 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:56.133387 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:50:56.133540 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:56.134118 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:56.135767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:50:56.137528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:00.793426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:00.795354 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:00.831649 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:00.832539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.832817 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.833646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.834156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:00.975740 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:00.977020 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:00.977365 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:00.977653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.978663 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.980436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.981188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:00.981797 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:00.982509 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:00.983044 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:00.983327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.984082 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:00.985997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.986496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:00.987711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:00.988167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:00.988655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:00.988655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:00.988655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:01.132999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:01.133343 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:01.134443 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:01.134956 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:01.135216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:01.136219 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:01.137342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:01.137790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:05.794216 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:05.795866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:05.827173 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:05.827656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.829305 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.830643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.831538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:05.976411 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:05.978371 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:05.978668 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:05.979039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.980445 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.982081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.983227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:05.983856 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:05.983856 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:05.984514 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:05.984784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.985560 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:05.987083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.987600 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:05.989402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:05.989960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:05.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:05.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:05.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:06.133661 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:06.134365 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:06.135127 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:06.135705 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:06.136117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:06.137287 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:06.139291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:06.139925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:10.794675 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:10.796118 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:10.832087 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:10.832445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.833456 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.834726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.835306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:10.977143 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:10.978282 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:10.978589 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:10.978888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.979683 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.981432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.982439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:10.983201 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:10.983458 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:10.983974 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:10.984243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:10.986378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:10.987473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:10.987975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:10.988787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:10.988787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:10.988787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:11.134218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:11.135035 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:11.135630 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:11.135881 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:11.136003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:11.136681 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:11.138615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:11.139071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:15.795541 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:15.796915 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:15.830334 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:15.830944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.831994 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.833325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.834557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:15.978098 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:15.979531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:15.979969 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:15.980530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.981245 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.983142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.983942 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:15.984711 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:15.984711 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:15.984711 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:15.984828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.985317 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:15.986244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.986842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:15.987267 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:15.987718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:15.988280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:15.988280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:15.988280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:16.134936 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:16.135730 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:16.136610 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:16.137123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:16.137378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:16.138609 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:16.140606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:16.141773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:20.796140 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:20.797987 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:20.810813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=352 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:51:20.839906 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:20.840187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.841078 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.841828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.842253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:20.978867 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:20.980149 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:20.980439 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:20.980661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.981677 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.983308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.983820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:20.984273 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:20.984612 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:20.985151 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:20.985427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.986276 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:20.987664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.988401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:20.989426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:20.990206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:20.991030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:20.991030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:20.991030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:21.135831 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:21.136438 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:21.137025 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:21.137291 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:21.137451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:21.138053 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:21.138941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:21.139461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:25.797023 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:25.798363 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:25.829297 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:25.829881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.830776 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.831636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.832236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:25.979588 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:25.981133 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:25.981476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:25.981871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.983157 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.984369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.984801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:25.985138 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:25.985626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:25.986750 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:25.987630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.988352 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:25.989387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.990193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:25.990886 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:25.991399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:25.991928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:25.991928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:25.991928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:26.136588 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:26.137063 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:26.137829 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:26.138260 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:26.138514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:26.139459 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:26.140639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:26.141139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:30.797985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:30.799742 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:30.830953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:30.831362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.832329 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.833317 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.834569 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:30.980418 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:30.981926 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:30.982484 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:30.983031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.984046 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.986107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.986508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:30.986857 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:30.986888 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:30.987416 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:30.987754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.988444 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:30.989480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.989833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:30.990804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:30.991150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:30.991690 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:30.991690 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:30.991690 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:31.137284 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:31.137918 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:31.138966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:31.138966 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:31.138966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:31.139513 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:31.140559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:31.141537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:35.798842 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:35.799957 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:35.836325 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:35.836812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.837671 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.838915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.839931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:35.981124 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:35.982551 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:35.982855 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:35.983155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.983984 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.985904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.986792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:35.987581 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:35.987581 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:35.987912 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:35.988104 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.988643 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:35.989739 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.990511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:35.990919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:35.991445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:35.992088 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:35.992088 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:35.992088 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:36.138210 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:36.138585 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:36.138988 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:36.139249 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:36.139397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:36.140038 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:36.141677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:36.142933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:40.799475 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:40.801569 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:40.841175 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:40.841591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.842573 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.843582 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.844584 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:40.981863 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:40.983075 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:40.983542 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:40.984176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.984881 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.986311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.987194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:40.988052 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:40.988348 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:40.988794 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:40.989040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.989687 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:40.991079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.991992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:40.992751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:40.993241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:40.993804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:40.993804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:40.993804 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:41.138935 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:41.139272 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:41.139835 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:41.140145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:41.140286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:41.141091 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:41.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:41.144002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:45.800170 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:45.801916 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:45.838901 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:45.839554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.840391 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.841378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.841902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:45.982687 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:45.984128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:45.984462 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:45.984783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.985840 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.987646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.988723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:45.989782 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:45.989782 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:45.989782 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:45.989782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.990338 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:45.991182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.991819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:45.992134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:45.992619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:45.993136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:45.993136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:45.993136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:46.139528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:46.139942 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:46.140417 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:46.140833 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:46.141105 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:46.141993 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:46.143649 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:46.144549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:50.800843 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:50.802576 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:50.834256 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:50.834793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.835341 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.836129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.836605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:50.983586 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:50.984884 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:50.985240 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:50.985512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.986785 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.988586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.990654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.991611 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:50.992507 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.993249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:50.994043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:50.995211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:50.996611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:50.996611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:50.996611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:51.140236 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:51.140843 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:51.141562 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:51.142016 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:51.142331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:51.143220 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:51.145064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:51.145983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:55.801890 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:55.803969 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:55.835489 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:55.835835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.836668 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.837495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.838346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:55.984546 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:55.985948 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:55.986318 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:55.986557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.987546 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.989448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.990229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:55.991318 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:55.991318 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:51:55.991318 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:55.991318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.991643 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:51:55.992462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.993209 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:51:55.993755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:55.994447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:55.995184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:55.995184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:55.995184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:51:56.141149 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:51:56.141552 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:51:56.142243 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:56.142611 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:51:56.142878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:56.143759 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:56.144955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:51:56.145921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:00.802925 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:00.805017 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:00.840571 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:00.841023 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.841915 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.844016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:00.985272 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:00.986968 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:00.987306 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:00.987746 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.988523 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.990916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.992167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:00.993331 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:00.993331 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:00.993331 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:00.993603 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.994313 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:00.995950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.996560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:00.998067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:00.998658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:00.999339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:00.999339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:00.999339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:01.141945 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:01.142812 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:01.143498 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:01.144012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:01.144361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:01.145589 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:01.147717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:01.148581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:05.803718 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:05.805191 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:05.840219 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:05.840716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.841676 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.842840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.843672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:05.985956 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:05.986942 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:05.987114 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:05.987355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.988374 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.990433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.991381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:05.992245 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:05.992654 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:05.993176 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:05.993419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.994034 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:05.995502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.996450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:05.997169 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:05.997759 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:05.998265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:05.998265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:05.998265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:06.142904 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:06.143635 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:06.144343 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:06.144931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:06.145229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:06.146294 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:06.148375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:06.149294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:10.804432 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:10.806738 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:10.856784 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:10.857162 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.858093 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.859294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.859843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:10.986575 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:10.987674 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:10.987952 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:10.988173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.989026 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.990905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.991635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:10.992482 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:10.992933 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:10.993297 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:10.993813 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.994430 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:10.995884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.996381 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:10.997836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:10.998289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:10.998795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:10.998795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:10.998795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:11.144006 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:11.144723 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:11.145514 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:11.146039 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:11.146346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:11.147448 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:11.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:11.150453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:15.805405 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:15.806674 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:15.839494 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:15.839754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.840411 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:15.841571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.842487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:15.987184 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:15.989724 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:15.990170 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:15.990398 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.991382 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:15.993431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.994060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:15.994530 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:15.994530 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:15.994617 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:15.994878 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.995567 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:15.997248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.997567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:15.999329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:16.000017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:16.000909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:16.000909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:16.000909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:16.144873 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:16.145435 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:16.145947 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:16.146401 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:16.146613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:16.147565 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:16.149654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:16.150478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:20.805971 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:20.807668 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:20.817298 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=240 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:52:20.842510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:20.842781 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.843429 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.844509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.845529 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:20.987871 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:20.988990 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:20.989313 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:20.989537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.990396 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.991524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.992428 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:20.994054 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:20.994276 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:20.994858 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:20.995156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.995897 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:20.996639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.997439 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:20.997910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:20.998834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:20.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:20.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:20.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:21.145423 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:21.146284 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:21.146936 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:21.147352 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:21.147543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:21.148547 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:21.150611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:21.151473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:25.806840 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:25.807866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:25.839345 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:25.839658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.840562 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:25.841747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.843205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:25.988476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:25.990035 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:25.990389 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:25.990641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.991612 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:25.993414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.994220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:25.995004 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:25.995083 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:25.995675 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:25.996143 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.997119 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:25.998467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:25.999193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:26.000523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:26.000993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:26.001558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:26.001558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:26.001558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:26.146340 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:26.147333 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:26.148385 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:26.149023 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:26.149282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:26.150407 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:26.152394 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:26.152926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:30.807335 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:30.808583 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:30.843423 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:30.843978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.844984 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.846199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.847116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:30.989296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:30.991107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:30.991665 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:30.991970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.992939 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.994279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.995173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:30.996074 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:30.996074 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:30.996074 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:30.996074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.996943 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:30.997713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.998276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:30.998677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:30.999192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:30.999875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:30.999875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:30.999875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:31.147067 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:31.148419 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:31.148999 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:31.149605 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:31.149605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:31.150326 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:31.152581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:31.153566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:35.808352 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:35.809964 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:35.847960 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:35.848347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.849239 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:35.850174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.850967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:35.990145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:35.991582 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:35.992340 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:35.992459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.993371 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:35.995022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.995863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:35.997034 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:35.997034 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:35.997034 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:35.997034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.998091 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:35.998902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.999386 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:35.999885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:36.000436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:36.001002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:36.001002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:36.001002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:36.148416 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:36.148944 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:36.149642 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:36.150065 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:36.150275 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:36.151079 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:36.152298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:36.153454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:40.817626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:40.818778 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:40.863418 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:40.863864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.864865 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:40.866071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.866905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:40.990893 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:40.992197 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:40.992541 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:40.992823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.993777 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:40.995599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.996339 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:40.997085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:40.997085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:40.997085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:40.997085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.997718 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:40.999096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:40.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:41.001071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:41.001477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:41.002014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:41.002014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:41.002014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:41.148962 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:41.149875 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:41.150593 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:41.150970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:41.151156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:41.151854 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:41.154096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:41.154940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:45.818312 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:45.820879 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:45.862416 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:45.862846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.863849 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:45.865347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.866298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:45.991534 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:45.992646 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:45.993023 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:45.993282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.994202 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:45.995465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.996374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:45.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:45.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:45.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:45.997543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.998244 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:45.999409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:45.999916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:46.001183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:46.001656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:46.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:46.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:46.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:46.149737 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:46.150523 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:46.151335 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:46.151794 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:46.152066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:46.153037 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:46.154935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:46.155909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:50.819024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:50.820581 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:50.849977 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:50.850468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.851418 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:50.852226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.853311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:50.992261 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:50.993471 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:50.993875 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:50.994204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.995066 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:50.997067 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:50.997990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:50.998931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:50.998931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:50.999119 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:50.999524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.000456 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:51.001644 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.002340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:51.003390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:51.003876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:51.004407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:51.004407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.004407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:51.150558 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:51.151034 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:51.151845 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:51.152389 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:51.152780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.153727 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:51.155306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:51.156201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:55.819946 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:55.821327 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:55.863987 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:55.864230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.865028 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:55.866126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.867623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:55.992945 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:55.994344 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:55.994839 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:55.995238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.996111 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:55.997423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:55.997933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:55.998473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:55.998833 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:52:55.999347 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:55.999650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.000415 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:52:56.001647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.002222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:52:56.003026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:56.003521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:56.004048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:56.004048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.004048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:52:56.151246 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:52:56.151903 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:52:56.152503 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:56.153019 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:52:56.153304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.154318 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:56.155391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:52:56.156355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:00.820487 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:00.822006 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:00.861467 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:00.862134 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:00.862935 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:00.864601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:00.865539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:00.993678 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:00.995147 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:00.995597 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:00.996008 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:00.997336 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:00.999320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.000375 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.001195 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:01.002161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:01.003628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:01.004416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:01.152172 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:01.152801 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:01.153206 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:01.153959 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:01.153959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.154275 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.155797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:01.156581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:05.821166 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:05.822960 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:05.858526 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:05.858812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.859873 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:05.860789 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.861731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:05.994194 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:05.995279 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:05.995561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:05.995856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.996738 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:05.998453 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:05.999537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:06.000642 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:06.000642 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:06.001120 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:06.001389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.002215 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:06.003872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.004378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:06.006336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:06.006859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:06.007442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:06.007442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.007442 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:06.152930 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:06.153650 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:06.154230 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:06.154641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:06.155093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.155780 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:06.157357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:06.158125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:10.822177 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:10.824448 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:10.856478 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:10.856934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:10.857812 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:10.858835 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:10.859922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:10.994772 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:10.996037 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:10.996287 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:10.996456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:10.997327 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:10.999671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.000967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:11.001988 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:11.001988 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:11.001988 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:11.001988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.002509 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:11.003827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.004568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:11.005367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:11.006020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:11.006687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:11.006687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.006687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:11.153672 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:11.154194 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:11.155069 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:11.155383 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:11.155618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.156275 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.157421 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:11.158016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:15.822761 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:15.824139 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:15.867173 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:15.867610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:15.868712 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:15.870217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:15.870935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:15.995504 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:15.997070 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:15.997540 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:15.997873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:15.998883 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:16.000503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.001124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:16.001784 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:16.001784 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:16.002210 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:16.002541 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.003300 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:16.004539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.005216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:16.006392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:16.007025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:16.007565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:16.007565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.007565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:16.154429 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:16.155469 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:16.156276 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:16.156714 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:16.157025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.158005 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.159912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:16.161289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:20.823472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:20.825178 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:20.826962 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=237 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:53:20.862871 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:20.863683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.864496 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:20.867462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.868556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:20.996115 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:20.996836 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:20.996963 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:20.997076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.997607 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:20.998384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:20.998841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:21.001586 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:21.001958 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:21.002292 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:21.002459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.003167 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:21.003922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.004425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:21.004750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:21.005306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:21.005866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:21.005866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.005866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:21.155472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:21.156189 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:21.156909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:21.157362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:21.157566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.158396 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:21.159727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:21.160612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:25.824165 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:25.825528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:25.857126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:25.857546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:25.858397 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:25.859281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:25.860270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:25.996755 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:25.998207 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:25.998524 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:25.998821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:25.999911 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:26.001133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.001652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:26.002321 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:26.002650 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:26.003366 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:26.003639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.004368 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:26.005204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.005995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:26.006724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.007592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:26.156396 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:26.156871 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:26.157666 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:26.158126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:26.158365 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.159650 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.161389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:26.162491 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:30.824833 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:30.826663 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:30.863765 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:30.864055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:30.864777 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:30.866180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:30.867117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:30.997422 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:30.998780 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:30.999020 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:30.999268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.000510 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:31.002200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.002761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:31.003328 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:31.003328 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:31.003854 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:31.004151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.005095 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:31.006511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.007050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:31.008108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:31.008597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:31.009405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:31.009405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.009405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:31.157056 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:31.157502 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:31.158254 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:31.158775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:31.159035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.160458 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.163107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:31.163676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:35.825492 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:35.827353 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:35.865176 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:35.865713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:35.866713 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:35.867643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:35.869069 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:35.998082 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:35.999336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:35.999709 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:36.000123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.001163 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:36.002953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.004041 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:36.005168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:36.005168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:36.005168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:36.005451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.006518 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:36.008207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.009452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:36.010598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:36.011260 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:36.011826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:36.011826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.011826 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:36.157822 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:36.158303 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:36.159052 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:36.159449 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:36.159723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.160624 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.162723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:36.163572 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:40.826244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:40.827833 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:40.862664 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:40.863059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:40.863914 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:40.865092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:40.865637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:40.998729 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:41.000148 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:41.000474 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:41.000626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.001593 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:41.003607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.004396 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:41.005263 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:41.005263 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:41.005263 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:41.005263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:41.007302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.008044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:41.009017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:41.009468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:41.010206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:41.010206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.010206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:41.158421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:41.159123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:41.159970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:41.160403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:41.160655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.161713 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.164281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:41.165176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:45.826914 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:45.828678 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:45.870981 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:45.871290 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:45.872254 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:45.873516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:45.874329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:45.999649 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:46.001052 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:46.001373 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:46.001641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.003202 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.004904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.005605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:46.006085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:46.006085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:46.006265 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:46.006454 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.007061 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:46.008242 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.008653 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:46.009341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:46.009815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:46.010306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:46.010306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.010306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:46.171336 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:46.171833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:46.172395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:46.172919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:46.172919 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:50.827605 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:50.829000 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:50.865481 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:50.865833 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:50.866827 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:50.868727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:50.869824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:51.000531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:51.002239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:51.002647 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:51.003029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.004225 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.006509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.007283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:51.007885 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:51.007885 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:51.008260 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:51.008656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.009482 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:51.010972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.011387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:51.012941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:51.013380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:51.013863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:51.013863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.013863 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:51.159897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:51.160933 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:51.161827 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:51.162324 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:51.162593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.163895 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.166378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:51.167356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:55.828421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:55.829678 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:55.863489 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:55.863963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:55.865232 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:55.866430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:55.867838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:56.001323 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:56.002730 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:56.003038 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:56.003273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.004249 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.005899 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.006948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:56.007896 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:56.008837 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:53:56.009445 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:56.009869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.010724 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:53:56.011757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:56.015144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:56.016117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.016992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:53:56.160647 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:53:56.161471 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:53:56.162267 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:56.162824 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:53:56.163145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.164228 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.165624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:53:56.166276 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:00.829495 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:00.831576 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:00.878045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:00.878045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:00.878589 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:00.880165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:00.881017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:01.002469 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:01.004017 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:01.004424 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:01.004713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.005823 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.007506 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.008602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:01.009446 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:01.009446 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:01.009866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:01.010199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.011193 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:01.013057 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.013838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:01.014876 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:01.015639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:01.016478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:01.016478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.016478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:01.161907 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:01.162725 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:01.164258 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:01.164537 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:01.164731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Traceback (most recent call last): Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.165623 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c9a59558-fce3-410a-8175-18c40e0a2091 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.166807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:01.167371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:05.830169 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:05.831595 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:05.870588 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:05.870933 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:05.871618 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:05.872401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:05.873341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:06.003033 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:06.004293 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:06.004464 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:06.004613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.005284 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.007243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.008054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:06.009152 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:06.009811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:06.010170 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:06.010323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.010855 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:06.012208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.013049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:06.013817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:06.014661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:06.015574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:06.015574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.015574 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:06.162535 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:06.163047 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:06.163533 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:06.163974 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:06.164216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.165139 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.167777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:06.168722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:10.831039 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:10.832813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:10.867498 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:10.868430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:10.869443 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:10.870393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:10.871288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:11.004171 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:11.005608 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:11.067944 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:11.067944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.067944 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:11.067944 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:11.067944 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:11.067944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.067944 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:11.068515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.069504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:11.070546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:11.071726 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.072788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:11.073297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:11.073803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.073803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:11.163295 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:11.163867 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:11.164388 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:11.164813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:11.165049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.165857 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.167620 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:11.168502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:15.832071 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:15.833471 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:15.869582 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:15.869840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:15.870211 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:15.871131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:15.872150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:16.004677 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:16.006055 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:16.006393 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:16.006639 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.007642 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:16.009641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.010783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:16.011897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:16.011897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:16.011897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:16.011897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.012405 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:16.013661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.014405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:16.015263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:16.016013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:16.016576 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:16.016576 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.016576 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:16.164136 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:16.164552 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:16.165424 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:16.165931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:16.166794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.167221 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.168915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:16.169718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:20.832499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:20.834181 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:20.836072 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=240 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:54:20.866307 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:20.866830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:20.867935 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:20.868738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:20.869589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:21.005386 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:21.006715 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:21.007063 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:21.007300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.008305 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:21.010224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.010725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:21.011244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:21.011244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:21.011483 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:21.011783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.012468 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:21.013222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.013677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:21.014163 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:21.014688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:21.015257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:21.015257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.015257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:21.164824 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:21.165396 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:21.166054 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:21.166290 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:21.166516 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.167398 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.169078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:21.170034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:25.833196 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:25.834577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:25.869561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:25.870012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:25.871072 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:25.872022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:25.873144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:26.006139 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:26.007447 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:26.007854 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:26.008348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.009431 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:26.011249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.011985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:26.012881 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:26.012881 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:26.012881 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:26.013106 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.013730 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:26.015237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.015775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:26.017406 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:26.017882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:26.018383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:26.018383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.018383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:26.165590 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:26.166281 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:26.166717 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:26.167117 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:26.167443 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.168413 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.170082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:26.171172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:30.833941 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:30.835371 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:30.872673 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:30.873029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:30.873331 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:30.874352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:30.875473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:31.039391 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:31.039549 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:31.040348 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:31.040853 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:31.041028 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:31.041185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.041670 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:31.041828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.042266 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.042637 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.043629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.044011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:31.044661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.045059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:31.045615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:31.045939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:31.046261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:31.046261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.046261 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:31.214463 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:31.215154 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:31.215566 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:31.215780 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:31.215897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.216405 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.218958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:31.219753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:35.834793 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:35.835690 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:35.868381 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:35.868782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:35.869467 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:35.870266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:35.870822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:36.040102 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:36.041137 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:36.042143 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:36.042649 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:36.042969 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:36.043233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.044236 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.046264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.047091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.047889 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:36.048973 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.049626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:36.050845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:36.051286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:36.051795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:36.051795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.051795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:36.215177 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:36.215939 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:36.216654 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:36.217125 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:36.217530 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.218401 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.220469 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:36.221387 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:40.835514 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:40.837055 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:40.899497 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:40.899882 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:40.900683 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:40.901544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:40.902542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:41.041059 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:41.042383 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:41.043499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:41.044105 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:41.044403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:41.044570 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.045291 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.045608 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:41.045928 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.047477 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:41.048055 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:54:41.048648 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:41.048648 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:41.048648 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:41.048648 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:41.048648 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:41.048648 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.048648 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:41.049007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.049495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:41.050284 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:41.050742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:41.051208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:41.051208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.051208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:41.215909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:41.216528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:41.217321 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:41.217866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:41.218119 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.219062 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.220794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:41.221527 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:45.836179 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:45.837938 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:45.873536 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:45.873881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:45.874534 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:45.875433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:45.876045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:46.042082 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:46.044792 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:46.045100 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:46.045986 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:46.046448 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:46.046900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.048457 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:46.049093 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:46.051587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.052568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.053855 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:46.054413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.055534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:46.056100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:46.057129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:46.058328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:46.058328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.058328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:46.216842 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:46.217280 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:46.217882 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:46.218307 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:46.218585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.219757 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.221436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:46.222466 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:50.837324 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:50.839237 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:50.881715 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:50.882185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:50.883416 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:50.884483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:50.885309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:51.043361 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:51.045076 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:51.045613 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:51.045962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.047257 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.047522 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:51.049083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.050040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:51.050723 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:51.050723 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:51.050723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.050881 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:51.052232 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:51.054052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:51.054645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:51.055337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:51.055337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.055337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:51.217648 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:51.218623 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:51.219468 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:51.219822 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:51.219981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.220818 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.222958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:51.223450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:55.839311 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:55.840499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:55.874777 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:55.875226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:55.876062 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:55.876953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:55.878274 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:56.044159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:56.045606 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:56.045984 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:56.046236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.047191 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:56.049124 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.049686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:56.050281 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:56.050281 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:54:56.050362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:56.050711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.051589 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:54:56.053076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:56.054870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:56.055719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.056937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:54:56.218494 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:54:56.219095 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:54:56.219507 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:56.219804 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:54:56.220182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.220785 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.222487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:54:56.223652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:00.840033 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:00.841624 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:00.877085 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:00.877283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:00.877904 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:00.878708 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:00.879196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:01.045131 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:01.046755 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:01.046974 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:01.047072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Traceback (most recent call last): Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.048257 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-c4ed58d2-61a2-4423-a792-e0d39591cef1 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:01.050566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.051110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:01.051736 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:01.051736 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:01.052259 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:01.052455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Traceback (most recent call last): Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.053071 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:01.054543 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.055045 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-6942b522-0c61-4a0d-9de1-dfdb0aabb90a None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:01.056726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:01.057403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:01.058049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:01.058049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.058049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:01.219087 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:01.219717 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:01.220377 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:01.220933 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:01.221214 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.222188 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.223843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:01.224220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:05.840951 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:05.842636 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:05.880830 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:05.881419 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:05.882458 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:05.883389 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:05.884248 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:06.046013 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:06.047553 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:06.047866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:06.048054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.049374 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:06.051761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.052304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:06.052909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:06.052909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:06.053362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:06.053676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.054675 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:06.055598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.056179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:06.056613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:06.057145 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:06.057645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:06.057645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.057645 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:06.240298 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:06.240977 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:06.241718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:10.841638 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:10.843283 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:10.875741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:10.875971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:10.877074 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:10.878485 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:10.879538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:11.047021 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:11.047860 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:11.048133 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:11.048361 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.049091 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:11.049934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.050392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:11.052565 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:11.053001 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:11.053375 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:11.053560 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.054117 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:11.055463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.055996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:11.057430 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:11.057921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:11.058399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:11.058399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.058399 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:11.220415 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:11.220937 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:11.221422 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:11.222022 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:11.222156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.223043 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.224962 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:11.226184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:15.842841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:15.844632 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:15.884893 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:15.885176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:15.885931 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:15.887136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:15.887555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:16.047675 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:16.049072 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:16.049579 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:16.049777 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.050630 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.051859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.052949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:16.053904 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:16.053904 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:16.054261 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:16.054568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.055359 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:16.056956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.057489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:16.058722 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:16.059187 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:16.059674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:16.059674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.059674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:16.221047 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:16.221727 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:16.222447 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:16.223078 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:16.223266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.224148 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.225997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:16.226873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:20.640488 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:20.651838 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:387}} Feb 16 19:55:20.652063 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Number of inconsistencies found at create/update: networks=4, routers=1, router_ports=1 {{(pid=105470) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:368}} Feb 16 19:55:20.652150 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Fixing resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:55:20.686629 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, resource={'id': '1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'name': 'private', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['247b2a4f-27c3-48ff-8b5b-2a6e34d81c2d', '6aa6bb34-b778-4413-99e5-d4ac2fc182da'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:40:06Z', 'updated_at': '2026-02-16T19:40:09Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 59043}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.687117 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.687684 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ddd02391-cc87-498e-9977-c16d9842f554, external_ids_update=None, columns={'tag': 59043}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.687905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.688940 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Failed to fix resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.720675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:55:20.721257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.721661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:55:20.721661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.721661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:55:20.721661 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Fixing resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:55:20.751169 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, resource={'id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'name': 'public', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0ee58c77-b622-4ab4-adb2-400b1ab81596', '8d726240-563e-4fd4-b3c3-c3b0599bca6a'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-02-16T19:40:12Z', 'updated_at': '2026-02-16T19:40:21Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.751169 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '0', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.751454 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-baea1945-0145-4a27-8595-4a08d27c2335, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.751630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.752363 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Failed to fix resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.753094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:55:20.753362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.753646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:55:20.753646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.753646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:55:20.753646 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Fixing resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:55:20.776159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, resource={'id': '81fd018a-121a-4a03-98c1-081a81a969fa', 'name': 'admin_net', 'tenant_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'project_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['84859701-b3de-4f64-be17-a75ccc95c0e1'], 'standard_attr_id': 42, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:43:33Z', 'updated_at': '2026-02-16T19:43:34Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 13657}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.776358 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, col_values=(('external_ids', {'neutron:network_name': 'admin_net', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.776576 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a88b6311-c4bb-44c5-bbf6-b6312d3411d6, external_ids_update=None, columns={'tag': 13657}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.776646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.777130 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Failed to fix resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.777941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:55:20.778459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.779089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:55:20.779089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.779089 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:55:20.779089 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Fixing resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:55:20.806924 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, resource={'id': '5aec5241-dd49-4846-b0bd-f32c1db95ac1', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f9fb342f-813a-4388-a760-afa136d8ba8f'], 'standard_attr_id': 52, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:44:08Z', 'updated_at': '2026-02-16T19:44:10Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 10787}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.807276 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, col_values=(('external_ids', {'neutron:network_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.807527 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb21044d-f495-47d0-9ee3-c77fa26dd090, external_ids_update=None, columns={'tag': 10787}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.807614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.808447 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Failed to fix resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.809640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:55:20.810222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.810858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:55:20.810858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.810858 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:55:20.810858 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Fixing resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:55:20.843756 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=342 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:55:20.845655 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:20.850825 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:20.921436 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.923132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.925741 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.930071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.931456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:20.934296 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, resource={'id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'enable_snat': True}, 'gw_port_id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'description': 'Created by cluster-api-provider-openstack cluster magnum-system-kube-haue7', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'ip_address': '172.24.5.78', 'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596'}, {'ip_address': '2001:db8::319', 'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-02-16T19:44:11Z', 'updated_at': '2026-02-16T19:44:17Z', 'revision_number': 3, 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf'}, resource_type=routers, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.934526 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, columns={'external_ids': {'neutron:router_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.934782 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.934938 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:20.935064 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.935748 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:20.935992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Unable to update router 15d28432-bd1b-492c-b98d-eccecccacd15. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Failed to fix resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1605, in update_router Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 256, in __exit__ Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Feb 16 19:55:20.937480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 222, in force_reraise Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1539, in update_router Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:55:20.937903 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:55:20.938545 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Fixing resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 19:55:21.048952 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:21.050816 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.051223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.051475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.052561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, resource={'id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'name': '', 'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'project_id': '', 'mac_address': 'fa:16:3e:29:33:4b', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'device_owner': 'network:router_gateway', 'standard_attr_id': 57, 'fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000155677', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-02-16T19:44:12Z', 'updated_at': '2026-02-16T19:44:16Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.053364 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0ee58c77-b622-4ab4-adb2-400b1ab81596 8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'neutron:network_name': 'neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-15d28432-bd1b-492c-b98d-eccecccacd15'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::319/64', '172.24.5.78/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.053463 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=493a778d-8a03-4344-aca8-f6d87f149d26, lrouter_port=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.053900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.055161 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:21.057719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.058494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:21.059253 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Failed to fix resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1942, in update_router_port Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:21.060366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 19:55:21.060859 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.061342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 19:55:21.061342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.061342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 19:55:21.061342 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:21.061342 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Maintenance task: Synchronization completed (took 0.41 seconds) Feb 16 19:55:21.061561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.061617 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.062149 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:21.063066 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.063689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:21.064094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:21.064683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.065382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:21.065382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.065382 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:21.222028 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:21.222436 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:21.223159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.223617 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:21.223905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.224871 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.226941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:21.227922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:25.847249 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:25.848906 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:25.884431 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:25.884431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:25.884785 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:25.885913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:25.886686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:26.049604 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:26.050897 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:26.051081 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:26.051216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.052014 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:26.053887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.054436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:26.058764 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:26.059170 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:26.059568 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:26.059909 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.060660 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:26.061447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.062362 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:26.062890 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:26.063688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:26.064623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:26.064623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.064623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:26.222534 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:26.223143 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:26.223942 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:26.224405 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:26.224898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.225930 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.227795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:26.228884 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:30.847848 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:30.849474 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:30.882855 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:30.883123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:30.883941 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:30.885153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:30.885626 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:31.050324 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:31.051130 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:31.051323 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:31.051478 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.052171 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:31.053404 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.054150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:31.059246 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:31.059905 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:31.060479 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:31.060718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.061378 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:31.062808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.063328 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:31.064565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:31.065070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:31.065614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:31.065614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.065614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:31.223212 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:31.223579 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:31.224143 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:31.224510 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:31.224877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.226119 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.227730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:31.228632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:35.848595 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:35.850518 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:35.889248 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:35.889489 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:35.890123 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:35.891282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:35.892437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:36.050958 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:36.052321 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:36.052606 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:36.052904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.053888 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:36.055583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.056462 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:36.059773 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:36.060496 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:36.061071 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:36.061357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.062180 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:36.063597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.064012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:36.065202 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:36.065632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:36.066120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:36.066120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.066120 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:36.223841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:36.224482 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:36.225369 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:36.225877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:36.226337 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.227256 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.228963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:36.229670 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:40.849632 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:40.850460 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:40.884343 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:40.884463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:40.885352 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:40.886672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:40.887671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:41.051997 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:41.053617 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:41.054144 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:41.054431 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.055566 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:41.057581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.059097 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:41.060328 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:41.061111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:41.061642 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:41.062013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.062845 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:41.064222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.064589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:41.066048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:41.066410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:41.066817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:41.066817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.066817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:41.224720 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:41.225572 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:41.226219 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:41.226939 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:41.226939 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.227358 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.229255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:41.230165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:45.850185 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:45.851664 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:45.890330 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:45.890798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:45.892240 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:45.892793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:45.893291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:46.052738 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:46.055437 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:46.055775 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:46.056015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.056888 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:46.058597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.059866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:46.060963 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:46.061358 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:46.061875 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:46.062205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.062916 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:46.063823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.064283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:46.064860 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:46.065384 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:46.065906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:46.065906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.065906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:46.225399 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:46.225982 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:46.226622 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:46.227168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:46.227456 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.228470 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.230343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:46.231575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:50.850799 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:50.852453 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:50.896825 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:50.897073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:50.897745 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:50.899225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:50.899790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:51.053453 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:51.054467 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:51.054752 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:51.054982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.056101 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:51.058025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.058555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:51.061341 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:51.061853 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:51.062212 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:51.062373 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.062831 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:51.063716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.064196 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:51.064714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:51.065183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:51.065814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:51.065814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.065814 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:51.226223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:51.226894 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:51.227631 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:51.228422 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:51.228422 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.229409 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.231126 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:51.232213 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:55.851679 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:55.853366 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:55.893225 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:55.893762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:55.894285 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:55.895501 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:55.896608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:56.054207 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:56.055819 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:56.056223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:56.056505 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.057688 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:56.059633 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.060643 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:56.061902 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:56.062748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:55:56.063300 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:56.063494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:55:56.065726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.066370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:55:56.068258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:56.070611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:56.071262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:56.071262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.071262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:55:56.227000 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:55:56.227801 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:55:56.228339 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:56.228557 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:55:56.228683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.229243 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.231007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:55:56.231378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:00.852093 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:00.853734 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:00.887858 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:00.888162 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:00.888854 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:00.889821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:00.890420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:01.054924 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:01.056064 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:01.056306 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:01.056472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.057147 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:01.058308 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.059094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:01.063022 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:01.063022 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:01.063437 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:01.063675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.064579 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:01.065671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:01.067311 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:01.067821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.068332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:01.227948 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:01.228512 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:01.229220 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:01.229640 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:01.229915 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.230773 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.231940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:01.232472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:05.852852 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:05.854651 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:05.893831 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:05.894221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:05.895138 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:05.896029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:05.896978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:06.055559 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:06.057096 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:06.057638 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:06.057980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.059035 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:06.060880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.061824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:06.062953 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:06.063468 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:06.064101 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:06.064407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.065318 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:06.066590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.067131 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:06.068470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:06.068948 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:06.069471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:06.069471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.069471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:06.228597 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:06.229374 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:06.230086 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:06.230588 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:06.230958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.231870 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.233544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:06.234390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:10.853585 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:10.855610 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:10.900588 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:10.900885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:10.901623 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:10.902888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:10.903817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:11.056323 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:11.057667 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:11.057998 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:11.058233 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.059255 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:11.061094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.061934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:11.063500 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:11.064156 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:11.064736 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:11.065054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.065900 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:11.067366 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.068022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:11.069085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:11.069786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:11.070295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:11.070295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.070295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:11.229355 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:11.230013 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:11.230597 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:11.231102 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:11.231349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.232221 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.233901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:11.235048 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:15.854385 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:15.855932 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:15.898147 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:15.898326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:15.899049 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:15.900013 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:15.901319 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:16.057194 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:16.058159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:16.058501 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:16.058681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.059740 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:16.061367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.062040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:16.063884 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:16.064811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:16.065628 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:16.065839 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.066552 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:16.068230 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.068887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:16.070109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:16.070770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:16.071474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:16.071474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.071474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:16.230183 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:16.231227 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:16.231962 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:16.232351 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:16.232589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.233772 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.235687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:16.236877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:20.855346 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:20.857253 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:20.858942 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=332 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:56:20.892310 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:20.892618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:20.893518 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:20.894680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:20.895235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:21.057745 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:21.059244 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:21.059590 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:21.059872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.060929 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:21.062943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.063856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:21.064770 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:21.065031 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:21.065430 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:21.065608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.066197 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:21.067532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.068081 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:21.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:21.069914 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:21.070451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:21.070451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.070451 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:21.231027 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:21.231402 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:21.232023 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:21.232350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:21.232602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.233390 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.234834 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:21.235333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:25.856255 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:25.858057 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:25.894128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:25.894359 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:25.895119 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:25.896425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:25.897094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:26.058609 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:26.060248 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:26.060557 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:26.060736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.061541 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:26.063457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.064448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:26.065399 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:26.065399 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:26.065673 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:26.065873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.066511 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:26.067892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.068614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:26.069498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:26.070408 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:26.071323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:26.071323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.071323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:26.231653 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:26.232356 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:26.233123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:26.233529 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:26.233812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.234624 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.236470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:26.237300 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:30.857190 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:30.858852 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:30.889886 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:30.890253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:30.891070 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:30.892111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:30.893108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:31.059290 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:31.060759 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:31.061169 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:31.061521 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.062534 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:31.064559 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.065046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:31.065555 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:31.065787 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:31.066293 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:31.066601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.067405 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:31.068685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.069099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:31.070594 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:31.071039 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:31.071539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:31.071539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.071539 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:31.232367 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:31.232952 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:31.233841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:31.234261 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:31.234554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.235578 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.237896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:31.238481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:35.857955 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:35.859574 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:35.894811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:35.895285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:35.895943 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:35.897127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:35.898053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:36.060055 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:36.061142 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:36.061440 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:36.061684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.062723 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:36.063963 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.064730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:36.065679 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:36.066381 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:36.066821 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:36.067016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.067629 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:36.069415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.069956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:36.071638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:36.072130 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:36.072637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:36.072637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.072637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:36.233086 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:36.233636 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:36.234473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:36.234910 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:36.235153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.236035 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.237532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:36.237894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:40.858644 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:40.860501 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:40.894426 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:40.895014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:40.895715 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:40.896597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:40.897098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:41.060689 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:41.062114 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:41.062425 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:41.062680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.063809 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:41.065652 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.066531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:41.067434 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:41.067434 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:41.067434 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:41.067434 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.067740 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:41.068979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.069840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:41.070871 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:41.071877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:41.073101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:41.073101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.073101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:41.233799 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:41.234377 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:41.235019 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:41.235462 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:41.235794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.236830 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.238465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:41.239614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:45.859412 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:45.861089 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:45.894813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:45.895236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:45.896295 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:45.898435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:45.899037 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:46.061908 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:46.063263 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:46.063559 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:46.063727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.064622 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:46.066836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.067811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:46.068549 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:46.068549 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:46.068549 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:46.068549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.069147 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:46.070802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.071280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:46.072712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:46.073193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:46.073716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:46.073716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.073716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:46.234584 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:46.235221 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:46.235731 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:46.236150 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:46.236418 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.237188 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.239332 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:46.240444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:50.860266 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:50.861974 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:50.895936 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:50.896135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:50.896613 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:50.897525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:50.898413 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:51.062548 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:51.063842 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:51.064211 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:51.064465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.065637 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:51.067563 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.068885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:51.069980 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:51.069980 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:51.069980 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:51.069980 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.070292 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:51.071636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.072660 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:51.073599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:51.074099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:51.074841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:51.074841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.074841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:51.235322 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:51.235798 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:51.236608 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:51.237087 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:51.237349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.238278 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.239954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:51.241144 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:55.861069 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:55.862971 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:55.897832 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:55.898287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:55.899281 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:55.900585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:55.901181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:56.063304 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:56.064721 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:56.065127 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:56.065380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.066423 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:56.068199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.068987 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:56.069831 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:56.069831 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:56:56.069831 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:56.070014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.070842 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:56:56.071971 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.072762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:56:56.073218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:56.073985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:56.074681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:56.074681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.074681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:56:56.235948 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:56:56.236424 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:56:56.237053 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:56.237326 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:56:56.237490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.238122 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.239392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:56:56.239945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:00.861857 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:00.863617 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:00.895467 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:00.895767 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:00.896677 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:00.897599 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:00.898564 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:01.063973 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:01.065443 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:01.065886 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:01.066155 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.067727 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:01.069577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.070474 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:01.071388 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:01.071388 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:01.071388 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:01.071388 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.071801 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:01.073082 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.073524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:01.074751 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:01.075186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:01.075686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:01.075686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.075686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:01.236563 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:01.237249 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:01.237918 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:01.238303 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:01.238546 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.239464 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.241336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:01.242307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:05.862896 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:05.864295 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:05.900076 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:05.900832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:05.901776 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:05.902736 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:05.903320 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:06.064811 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:06.066218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:06.066635 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:06.066925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.067850 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:06.069551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.070661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:06.071587 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:06.071587 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:06.071587 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:06.071821 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.072731 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:06.074002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.074846 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:06.075593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:06.076391 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:06.077360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:06.077360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.077360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:06.237272 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:06.237891 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:06.238389 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:06.238883 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:06.239277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.239940 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.241592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:06.242410 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:10.863464 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:10.864660 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:10.902255 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:10.902865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:10.904824 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:10.905602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:10.906586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:11.065583 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:11.067069 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:11.067351 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:11.067656 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.068815 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:11.070801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.071550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:11.072373 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:11.072373 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:11.072373 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:11.072601 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.073561 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:11.075003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.075733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:11.076799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:11.077412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:11.078003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:11.078003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.078003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:11.237989 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:11.238728 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:11.239407 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:11.239866 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:11.240118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.241166 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.243192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:11.243740 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:15.864163 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:15.865958 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:15.905805 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:15.905978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:15.906648 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:15.907676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:15.908589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:16.066294 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:16.067977 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:16.068429 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:16.068782 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.069988 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:16.072022 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.073077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:16.073931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:16.073931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:16.073931 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:16.073931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.074186 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:16.075476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.075934 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:16.077192 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:16.077641 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:16.078178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:16.078178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.078178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:16.238666 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:16.239763 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:16.240248 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:16.240526 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:16.240787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.242238 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.244370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:16.245605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:20.865517 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:20.867144 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:20.868524 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=240 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:57:20.898532 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:20.898970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:20.899754 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:20.901353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:20.902959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:21.067400 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:21.068880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:21.069128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:21.069291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.070371 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:21.072324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.073343 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.074949 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:21.076135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.076867 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:21.077794 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:21.079135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:21.079721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:21.079721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.079721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:21.239661 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:21.240547 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:21.241412 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:21.241887 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:21.241972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.242926 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.244727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:21.245243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:25.866268 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:25.867793 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:25.906877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:25.907247 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:25.908119 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:25.909376 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:25.910627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:26.068146 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:26.069569 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:26.069894 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:26.070203 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.071229 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:26.072296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.072996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:26.073678 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:26.074131 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:26.074650 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:26.074929 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.075561 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:26.076647 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.077076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:26.078098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:26.078630 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:26.079195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:26.079195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.079195 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:26.240399 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:26.241045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:26.241591 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:26.241987 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:26.242146 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.242808 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.244090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:26.244790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:30.867110 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:30.868335 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:30.902214 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:30.902459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:30.903253 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:30.904349 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:30.905498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:31.068907 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:31.069971 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:31.070324 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:31.070568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.071493 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:31.072411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.073325 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:31.074226 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:31.074877 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:31.075375 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:31.075727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.076494 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:31.077921 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.078798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:31.079672 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:31.080263 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:31.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:31.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:31.241111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:31.241843 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:31.242363 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:31.242946 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:31.243208 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.244225 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.245845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:31.246575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:35.867770 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:35.869406 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:35.909307 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:35.909575 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:35.910661 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:35.911596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:35.912523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:36.069328 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:36.071074 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:36.071434 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:36.071675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.072972 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:36.075025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.075815 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:36.076967 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:36.076967 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:36.076967 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:36.076967 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.077654 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:36.079288 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.080379 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:36.081054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:36.082092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:36.083299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:36.083299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.083299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:36.241860 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:36.242627 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:36.243258 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:36.243668 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:36.243985 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.244906 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.246676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:36.247502 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:40.868337 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:40.869957 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:40.908569 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:40.908829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:40.909437 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:40.910628 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:40.911738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:41.070138 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:41.071225 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:41.071606 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:41.071916 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.072998 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:41.074610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.075514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:41.076464 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:41.076549 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:41.077232 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:41.077593 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.078855 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:41.080198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.080684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:41.081978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:41.082447 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:41.083002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:41.083002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.083002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:41.242559 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:41.243264 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:41.244421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:41.244720 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:41.244979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.245961 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.248017 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:41.248787 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:45.868850 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:45.870418 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:45.908452 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:45.908932 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:45.909861 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:45.910801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:45.911986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:46.070987 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:46.072239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:46.072572 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:46.072850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.073817 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:46.075053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.075892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:46.077060 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:46.077168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:46.077724 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:46.078036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.078970 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:46.080571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.081110 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:46.082651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:46.083116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:46.083610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:46.083610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.083610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:46.243410 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:46.244173 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:46.244752 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:46.245039 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:46.245176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.245954 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.247718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:46.248522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:50.869799 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:50.871333 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:50.906102 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:50.906525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:50.907571 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:50.908873 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:50.910193 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:51.071721 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:51.073794 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:51.074123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:51.074356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.075378 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:51.077538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.078132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:51.078726 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:51.078726 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:51.078819 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:51.079156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.080268 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:51.081060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.081532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:51.082098 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:51.082580 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:51.083128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:51.083128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.083128 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:51.244158 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:51.245051 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:51.245843 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:51.246304 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:51.246565 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.247583 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.249586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:51.250470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:55.870447 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:55.871797 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:55.909462 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:55.909738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:55.911487 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:55.913943 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:55.914498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:56.072772 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:56.074410 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:56.074814 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:56.075077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.076216 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:56.078329 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.079159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:56.080223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:56.080223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:57:56.080223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:56.080223 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.081484 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:57:56.082476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.083139 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:57:56.083598 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:56.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:56.084862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:56.084862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.084862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:57:56.244748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:57:56.245489 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:57:56.246223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:56.246634 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:57:56.246913 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.247861 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.249618 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:57:56.250444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:00.871675 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:00.873361 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:00.909028 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:00.909467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:00.910460 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:00.912681 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:00.913875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:01.073501 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:01.074728 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:01.074963 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:01.075166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.076206 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:01.078059 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.078880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:01.079687 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:01.079687 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:01.079980 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:01.080255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.080872 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:01.081807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.082262 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:01.082880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:01.083347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:01.083885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:01.083885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.083885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:01.245441 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:01.246577 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:01.247183 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:01.247584 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:01.247877 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.249014 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.250775 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:01.251625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:05.872478 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:05.873961 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:05.910640 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:05.911194 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:05.912179 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:05.913083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:05.913947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:06.074210 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:06.075716 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:06.076017 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:06.076217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.076978 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:06.078556 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.079831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:06.080741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:06.080741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:06.080741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:06.081033 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.081990 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:06.083623 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.084256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:06.085470 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:06.086227 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:06.086982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:06.086982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.086982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:06.246176 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:06.246685 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:06.247452 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:06.247918 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:06.248118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.249035 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.251102 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:06.252356 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:10.873159 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:10.874850 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:10.906156 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:10.906515 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:10.907411 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:10.908271 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:10.909241 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:11.075688 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:11.076847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:11.077213 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:11.077511 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.078635 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:11.081282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.081800 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:11.082345 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:11.082345 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:11.082750 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:11.083632 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.084226 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:11.085205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:11.086191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:11.086637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.087164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:11.246947 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:11.247672 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:11.248272 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:11.248661 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:11.248996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.249926 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.251726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:11.252558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:15.873848 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:15.875319 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:15.909129 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:15.909830 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:15.910561 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:15.911935 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:15.912492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:16.076363 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:16.077569 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:16.077913 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:16.078152 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.079587 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:16.082060 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.082496 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:16.083111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:16.083155 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:16.084347 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:16.084347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.085121 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:16.085852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.086747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:16.087177 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:16.087953 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:16.088792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:16.088792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.088792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:16.247729 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:16.248481 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:16.249119 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:16.249535 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:16.249822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.250714 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.252433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:16.253363 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:20.874463 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:20.875923 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:20.877476 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=240 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:58:20.916414 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:20.916683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:20.917494 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:20.918654 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:20.919294 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:21.076974 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:21.078660 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:21.079107 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:21.079586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.080499 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:21.081861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.082629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:21.083499 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:21.084212 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:21.084539 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:21.084749 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.085342 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:21.086733 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.087455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:21.088432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:21.089255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:21.090101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:21.090101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.090101 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:21.248441 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:21.249213 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:21.249834 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:21.250223 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:21.250509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.251412 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.253138 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:21.253829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:25.875268 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:25.876936 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:25.913461 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:25.914029 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:25.914976 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:25.916185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:25.917042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:26.077817 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:26.078656 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:26.078814 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:26.079051 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.079444 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:26.080277 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.081165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:26.084160 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:26.084533 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:26.084992 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:26.085254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.085897 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:26.087297 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.087805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:26.089076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:26.089611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:26.090264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:26.090264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.090264 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:26.249239 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:26.249880 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:26.250569 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:26.251168 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:26.251273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.257436 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.259184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:26.259573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:30.876088 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:30.877971 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:30.912804 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:30.913229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:30.914205 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:30.915077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:30.916015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:31.078461 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:31.079574 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:31.079917 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:31.080135 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.080974 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:31.081954 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.082457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:31.084495 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:31.084863 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:31.085425 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:31.085425 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.085982 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:31.086865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.087416 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:31.087881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:31.088348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:31.088887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:31.088887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.088887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:31.249935 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:31.250490 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:31.250893 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:31.251188 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:31.251557 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.252123 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.253432 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:31.253823 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:35.876946 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:35.878568 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:35.917884 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:35.918112 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:35.919178 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:35.920240 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:35.921185 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:36.079127 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:36.080722 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:36.081309 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:36.081676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.082514 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:36.084438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.085142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:36.085735 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:36.085735 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:36.085735 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:36.085896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.086558 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:36.087168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:36.088002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:36.088790 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.089755 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:36.250785 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:36.251645 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:36.252248 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:36.252739 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:36.252988 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.253936 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.255836 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:36.257176 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:40.887394 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:40.890637 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:40.918170 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:40.918726 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:40.919725 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:40.920612 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:40.921857 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:41.079936 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:41.081734 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:41.082089 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:41.082306 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.083519 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:41.085273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.085851 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:41.086401 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:41.086474 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:41.086936 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:41.087198 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.087847 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:41.088805 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.089342 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:41.089945 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:41.090610 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:41.091160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:41.091160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.091160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:41.251397 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:41.252328 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:41.253337 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:41.253709 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:41.253989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.254955 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.257096 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:41.258026 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:45.887837 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:45.889464 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:45.924278 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:45.924801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:45.925820 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:45.926992 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:45.928015 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:46.080675 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:46.082118 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:46.082530 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:46.082984 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.084251 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:46.086435 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.087280 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:46.088251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:46.088251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:46.088251 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:46.088589 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.089552 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:46.090959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.091459 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:46.092544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:46.093011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:46.093550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:46.093550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.093550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:46.252134 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:46.252651 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:46.253450 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:46.253883 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:46.254595 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.255655 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.257896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:46.259076 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:50.888624 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:50.889896 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:50.925681 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:50.925986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:50.926721 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:50.928021 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:50.928588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:51.081360 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:51.082606 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:51.082986 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:51.083251 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.084285 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:51.085996 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.086912 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:51.087870 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:51.087870 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:51.088369 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:51.088663 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.089493 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:51.090925 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.091724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:51.092683 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:51.093243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:51.093881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:51.093881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.093881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:51.252858 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:51.253428 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:51.253889 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:51.254581 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:51.254581 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.255215 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.256159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:51.257161 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:55.889441 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:55.891129 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:55.924465 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:55.925024 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:55.925931 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:55.926831 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:55.927392 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:56.082013 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:56.083214 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:56.083812 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:56.083812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.084553 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:56.086525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.087369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:56.088229 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:56.088229 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:58:56.088229 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:56.088473 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.089183 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:58:56.089983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.090802 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:58:56.091249 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:56.092006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:56.092907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:56.092907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.092907 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:58:56.253594 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:58:56.254186 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:58:56.254718 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:56.255145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:58:56.255397 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.256367 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.258032 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:58:56.258902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:00.890345 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:00.892005 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:00.927543 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:00.927854 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:00.928829 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:00.930207 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:00.931166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:01.082851 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:01.083813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:01.083813 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:01.083813 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.084369 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:01.085370 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.086269 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:01.087808 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:01.088492 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:01.088710 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:01.088960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.089727 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:01.090738 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.091420 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:01.092182 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:01.092729 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:01.093237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:01.093237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.093237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:01.254215 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:01.254950 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:01.255486 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:01.255908 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:01.256111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.256965 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.258753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:01.259287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:05.890889 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:05.891950 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:05.928890 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:05.929180 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:05.929880 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:05.931035 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:05.931455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:06.083216 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:06.084359 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:06.084604 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:06.084965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.085891 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:06.087788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.088179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:06.088556 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:06.088939 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:06.089545 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:06.089956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.090874 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:06.091796 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.092330 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:06.092902 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:06.093415 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:06.093937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:06.093937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.093937 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:06.254997 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:06.255572 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:06.256049 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:06.256275 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:06.256497 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.257285 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.258803 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:06.259304 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:10.891524 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:10.892839 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:10.955270 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:10.955607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:10.956494 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:10.958116 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:10.959153 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:11.083846 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:11.084847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:11.085089 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:11.085243 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.085922 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:11.087150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.087812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:11.088947 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:11.089687 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:11.090130 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:11.090287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.090847 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:11.092052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.092646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:11.093433 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:11.093983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:11.094583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:11.094583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.094583 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:11.255924 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:11.256360 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:11.257867 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:11.258350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:11.258837 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.259958 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.261448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:11.262512 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:15.892370 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:15.894473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:15.930570 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:15.930969 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:15.931822 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:15.933003 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:15.933927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:16.084397 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:16.085835 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:16.086175 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:16.086440 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.087458 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:16.089761 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.090960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:16.091809 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:16.091809 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:16.091809 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:16.091809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.092225 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:16.093095 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:16.094077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:16.094754 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.095283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:16.256641 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:16.257060 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:16.257689 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:16.258166 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:16.258414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.259229 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.260346 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:16.261522 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:20.893153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:20.894876 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:20.896917 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=240 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 19:59:20.936731 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:20.937079 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:20.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:20.939040 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:20.939561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:21.085317 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:21.086544 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:21.086851 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:21.087034 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.087892 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:21.090027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.090896 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:21.091793 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:21.091793 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:21.092901 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:21.093125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.093837 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:21.095636 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.096586 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:21.097229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:21.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:21.098159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:21.098159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.098159 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:21.257145 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:21.257558 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:21.258315 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:21.258737 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:21.259007 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.259910 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.261142 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:21.261927 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:25.893856 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:25.895433 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:25.930982 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:25.931380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:25.932606 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:25.934019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:25.935149 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:26.085965 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:26.087253 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:26.087682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:26.087979 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.088901 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:26.089997 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.090429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:26.091302 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:26.091636 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:26.092183 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:26.092486 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.093256 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:26.094125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.094537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:26.095132 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:26.095684 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:26.096211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:26.096211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.096211 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:26.257853 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:26.258456 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:26.259385 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:26.260314 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:26.260314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.261355 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.263094 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:26.263949 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:30.894582 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:30.896356 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:30.936427 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:30.936906 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:30.937820 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:30.939270 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:30.939772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:31.086622 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:31.087614 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:31.087789 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:31.087982 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.088687 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:31.089651 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.090525 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:31.091726 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:31.092369 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:31.092814 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:31.092970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.093812 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:31.094542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:31.095657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:31.096380 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.097245 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:31.258410 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:31.259186 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:31.259951 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:31.260847 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:31.260847 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.261303 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.262524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:31.263031 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:35.895198 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:35.896935 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:35.930792 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:35.931791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:35.932597 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:35.933463 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:35.934347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:36.087264 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:36.088074 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:36.088286 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:36.088465 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.089127 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:36.090273 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.091111 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:36.092179 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:36.093128 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:36.093561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:36.093799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.094414 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:36.095889 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.097006 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:36.097664 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:36.098550 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:36.099503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:36.099503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.099503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:36.259446 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:36.260271 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:36.261020 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:36.261417 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:36.261602 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.262481 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.264378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:36.265455 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:40.935484 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:40.936801 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:40.963562 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:40.963865 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:40.964531 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:40.965426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:40.966534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:41.088299 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:41.089709 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:41.090043 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:41.090279 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.091266 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:41.093025 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.093838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:41.094734 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:41.094734 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:41.094734 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:41.094965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.095750 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:41.097171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.097665 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:41.098986 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:41.099482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:41.100016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:41.100016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.100016 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:41.260339 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:41.261287 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:41.262139 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:41.262563 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:41.262866 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.263883 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.265718 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:41.266567 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:45.936100 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:45.937709 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:45.970772 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:45.971164 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:45.971960 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:45.972741 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:45.973385 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:46.089130 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:46.090337 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:46.090559 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:46.090748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.091523 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:46.093377 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.094151 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:46.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:46.094969 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:46.095423 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:46.095832 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.096716 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:46.097950 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.098753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:46.099323 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:46.100282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:46.101220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:46.101220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.101220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:46.261218 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:46.261733 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:46.262212 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:46.262626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:46.262875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.263686 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.265318 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:46.266179 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:50.937183 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:50.939362 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:50.970827 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:50.971282 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:50.971917 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:50.972766 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:50.973941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:51.089970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:51.091415 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:51.091817 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:51.092084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.093197 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:51.095183 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.095983 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:51.096865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:51.096865 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:51.097472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:51.097671 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Traceback (most recent call last): Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.098533 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:51.099881 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.100841 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-bd9cfc25-d044-4118-b345-916282a31221 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:51.102166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:51.102922 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:51.103472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:51.103472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.103472 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:51.261857 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:51.262582 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:51.263209 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:51.263716 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:51.264011 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.264989 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.266727 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:51.267611 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:55.938103 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:55.939885 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:55.977531 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:55.977837 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:55.978551 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:55.979786 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:55.980676 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:56.090599 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:56.092364 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:56.092885 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:56.093235 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.094528 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:56.097156 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.098085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:56.099045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:56.099045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 19:59:56.099045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:56.099278 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Traceback (most recent call last): Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.100009 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 19:59:56.100941 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.101426 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-1c5b918f-f09a-4d3d-b385-153fc709a4cc None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 19:59:56.102109 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:56.102592 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:56.103150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:56.103150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.103150 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 19:59:56.262511 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 19:59:56.263175 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 19:59:56.263710 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:56.264152 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 19:59:56.264402 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Traceback (most recent call last): Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.265305 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-5feb275c-ef7e-4f20-a6ea-ad15e0af86d5 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.267036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 19:59:56.267558 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:00.938646 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:00.939937 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:00.980612 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:00.981047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:00.981923 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:00.982792 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:00.983498 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:01.091369 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:01.093046 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:01.093319 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:01.093588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.094500 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:01.096638 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.097302 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:01.098088 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:01.098583 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:01.099135 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:01.099480 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.100296 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:01.101614 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.102125 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:01.103295 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:01.103849 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:01.104551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:01.104551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.104551 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:01.263181 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:01.263573 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:01.264192 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:01.264467 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:01.264629 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.265371 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.267200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:01.268341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:05.939421 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:05.941265 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:05.976269 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:05.976674 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:05.977798 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:05.979053 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:05.979908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:06.094876 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:06.094876 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:06.094876 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:06.095482 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.095930 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:06.097038 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.097518 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:06.098648 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:06.099111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:06.099385 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:06.099579 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.100299 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:06.100993 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.101484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:06.102100 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:06.102785 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:06.103524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:06.103524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.103524 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:06.263673 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:06.264228 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:06.264791 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:06.265236 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:06.265532 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.266364 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.267166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:06.267646 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:10.940276 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:10.941985 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:10.981472 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:10.981894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:10.982838 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:10.983809 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:10.984900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:11.113682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:11.113682 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:11.115405 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:11.116597 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:11.117164 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:11.117471 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.118756 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.119475 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:11.119970 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:11.121619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.122568 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.123537 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.124627 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:11.125417 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:11.126265 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:11.127200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:11.127200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.127200 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:11.264663 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:11.265111 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:11.265981 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:11.266468 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:11.266820 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.267850 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.269774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:11.270827 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:15.941178 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:15.942687 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:15.977357 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:15.977657 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:15.978400 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:15.979542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:15.980312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:16.114156 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:16.114790 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:16.115561 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:16.137763 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:16.137763 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:16.137869 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.138619 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:16.139730 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.140141 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:16.140544 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:16.140544 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.140689 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:16.141395 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.141753 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:16.142154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:16.142509 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:16.142888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:16.142888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.142888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:16.265489 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:16.265889 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:16.266633 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:16.267025 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:16.267258 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.268085 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.269975 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:16.270666 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:20.641201 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:20.652012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:387}} Feb 16 20:00:20.652326 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Number of inconsistencies found at create/update: networks=4, routers=1, router_ports=1 {{(pid=105470) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:368}} Feb 16 20:00:20.652326 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Fixing resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 20:00:20.684928 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, resource={'id': '1e662dc7-bbd7-4f79-9523-34c3de2c8994', 'name': 'private', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['247b2a4f-27c3-48ff-8b5b-2a6e34d81c2d', '6aa6bb34-b778-4413-99e5-d4ac2fc182da'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:40:06Z', 'updated_at': '2026-02-16T19:40:09Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 59043}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.685292 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-1e662dc7-bbd7-4f79-9523-34c3de2c8994, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.685688 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ddd02391-cc87-498e-9977-c16d9842f554, external_ids_update=None, columns={'tag': 59043}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.685843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.686791 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Failed to fix resource 1e662dc7-bbd7-4f79-9523-34c3de2c8994 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.687895 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 20:00:20.688492 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.689458 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 20:00:20.689458 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.689458 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 20:00:20.689458 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Fixing resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 20:00:20.715164 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, resource={'id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'name': 'public', 'tenant_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'project_id': 'e81953b0c60d4ba9ab10a434ec8d841f', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0ee58c77-b622-4ab4-adb2-400b1ab81596', '8d726240-563e-4fd4-b3c3-c3b0599bca6a'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-02-16T19:40:12Z', 'updated_at': '2026-02-16T19:40:21Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.715579 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '0', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.716286 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-baea1945-0145-4a27-8595-4a08d27c2335, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.716624 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.717680 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Failed to fix resource c88d443e-2b28-4e97-8173-0ccd2ed29071 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.718811 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 20:00:20.719168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.719607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 20:00:20.719607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.719607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 20:00:20.719607 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Fixing resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 20:00:20.748830 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, resource={'id': '81fd018a-121a-4a03-98c1-081a81a969fa', 'name': 'admin_net', 'tenant_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'project_id': '971fc8fb2f5a4b77b59815eb5f88914a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['84859701-b3de-4f64-be17-a75ccc95c0e1'], 'standard_attr_id': 42, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:43:33Z', 'updated_at': '2026-02-16T19:43:34Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 13657}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.749095 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-81fd018a-121a-4a03-98c1-081a81a969fa, col_values=(('external_ids', {'neutron:network_name': 'admin_net', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.749424 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a88b6311-c4bb-44c5-bbf6-b6312d3411d6, external_ids_update=None, columns={'tag': 13657}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.749547 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.750191 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Failed to fix resource 81fd018a-121a-4a03-98c1-081a81a969fa (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.751204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 20:00:20.751533 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.751955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 20:00:20.751955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.751955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 20:00:20.751955 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Fixing resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 20:00:20.777147 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, resource={'id': '5aec5241-dd49-4846-b0bd-f32c1db95ac1', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f9fb342f-813a-4388-a760-afa136d8ba8f'], 'standard_attr_id': 52, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-02-16T19:44:08Z', 'updated_at': '2026-02-16T19:44:10Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 10787}, resource_type=networks, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.777558 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5aec5241-dd49-4846-b0bd-f32c1db95ac1, col_values=(('external_ids', {'neutron:network_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.778142 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb21044d-f495-47d0-9ee3-c77fa26dd090, external_ids_update=None, columns={'tag': 10787}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.778368 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.779206 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Failed to fix resource 5aec5241-dd49-4846-b0bd-f32c1db95ac1 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2298, in update_network Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.780073 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 20:00:20.780615 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.781257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 20:00:20.781257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.781257 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 20:00:20.781257 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Fixing resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 20:00:20.841116 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, resource={'id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'project_id': '8788d2932a2c446fb8c7daa2320f6ddf', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'enable_snat': True}, 'gw_port_id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'description': 'Created by cluster-api-provider-openstack cluster magnum-system-kube-haue7', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'external_fixed_ips': [{'ip_address': '172.24.5.78', 'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596'}, {'ip_address': '2001:db8::319', 'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-02-16T19:44:11Z', 'updated_at': '2026-02-16T19:44:17Z', 'revision_number': 3, 'tenant_id': '8788d2932a2c446fb8c7daa2320f6ddf'}, resource_type=routers, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.841548 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, columns={'external_ids': {'neutron:router_name': 'k8s-clusterapi-cluster-magnum-system-kube-haue7', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.842045 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.842319 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "493a778d-8a03-4344-aca8-f6d87f149d26"')], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.842561 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.843460 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:20.843875 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Unable to update router 15d28432-bd1b-492c-b98d-eccecccacd15. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Failed to fix resource 15d28432-bd1b-492c-b98d-eccecccacd15 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1605, in update_router Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 256, in __exit__ Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Feb 16 20:00:20.844818 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 222, in force_reraise Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1539, in update_router Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 20:00:20.845393 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 20:00:20.846093 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Fixing resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports) at create/update {{(pid=105470) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:397}} Feb 16 20:00:20.908550 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=410 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 20:00:20.942002 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:20.942717 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, resource={'id': '493a778d-8a03-4344-aca8-f6d87f149d26', 'name': '', 'network_id': 'c88d443e-2b28-4e97-8173-0ccd2ed29071', 'project_id': '', 'mac_address': 'fa:16:3e:29:33:4b', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '15d28432-bd1b-492c-b98d-eccecccacd15', 'device_owner': 'network:router_gateway', 'standard_attr_id': 57, 'fixed_ips': [{'subnet_id': '0ee58c77-b622-4ab4-adb2-400b1ab81596', 'ip_address': '172.24.5.78'}, {'subnet_id': '8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'ip_address': '2001:db8::319'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000155677', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-02-16T19:44:12Z', 'updated_at': '2026-02-16T19:44:16Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.943103 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0ee58c77-b622-4ab4-adb2-400b1ab81596 8d726240-563e-4fd4-b3c3-c3b0599bca6a', 'neutron:network_name': 'neutron-c88d443e-2b28-4e97-8173-0ccd2ed29071', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-15d28432-bd1b-492c-b98d-eccecccacd15'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::319/64', '172.24.5.78/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.943789 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=493a778d-8a03-4344-aca8-f6d87f149d26, lrouter_port=lrp-493a778d-8a03-4344-aca8-f6d87f149d26, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.944663 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.945333 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Failed to fix resource 493a778d-8a03-4344-aca8-f6d87f149d26 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 410, in check_for_inconsistencies Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 319, in _fix_create_update Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1942, in update_router_port Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.946799 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Feb 16 20:00:20.947344 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.947955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Feb 16 20:00:20.947955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.947955 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Feb 16 20:00:20.947955 np0000155677 neutron-ovn-maintenance-worker[105470]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Maintenance task: Synchronization completed (took 0.29 seconds) Feb 16 20:00:20.947955 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:20.986614 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:20.986947 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.987870 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.989199 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:20.989904 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:21.114870 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:21.115715 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:21.116509 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:21.117057 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:21.117438 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:21.117685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.118870 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:21.121118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.121588 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:21.122092 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:21.122092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.122481 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:21.123542 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.124259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:21.124959 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:21.125461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:21.126002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:21.126002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.126002 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:21.266038 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:21.266632 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:21.267430 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:21.267926 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:21.268204 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.269174 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.271019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:21.272481 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:25.944473 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:25.946286 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:25.984535 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:25.984991 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:25.985981 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:25.987071 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:25.987637 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:26.115524 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:26.116843 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:26.117998 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:26.119163 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:26.119528 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:26.119711 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.120655 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:26.120905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.121490 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:26.123314 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:26.125573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:26.126171 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:26.126872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:26.126872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.126872 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:26.127036 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.127577 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:26.266958 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:26.267292 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:26.268072 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:26.268533 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:26.268797 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.269870 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.270709 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:26.271108 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:30.945120 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:30.946970 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:30.993517 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:30.993900 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Traceback (most recent call last): Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:30.994822 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-17e57b68-bf1b-48b9-86d2-cc9a223b9f17 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:30.996118 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:30.996905 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:31.116160 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:31.117135 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:31.117439 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:31.117712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.118197 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:31.119037 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.119286 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:31.120548 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.121046 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:31.121564 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:31.121840 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.122423 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:31.123445 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.123885 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:31.124783 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:31.125236 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:31.125795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:31.125795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.125795 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:31.267599 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:31.267988 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:31.268909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:31.269322 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:31.269605 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.270549 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.272483 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:31.273438 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:35.946204 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:35.947762 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:35.987153 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:35.987457 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:35.988078 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:35.989123 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:35.989566 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:36.117260 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:36.118722 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:36.119100 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:36.119460 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.120181 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:36.121597 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.122407 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:36.123266 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:36.123476 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:36.126160 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.127117 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.128166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.129165 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:36.129844 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:36.130503 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:36.131336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:36.131336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.131336 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:36.268520 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:36.269123 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:36.270039 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:36.270482 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:36.270735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.271607 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.273744 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:36.274901 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:40.947188 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:40.949126 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:40.986827 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:40.987275 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:40.988234 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:40.989461 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:40.990347 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:41.118055 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:41.119096 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:41.119403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:41.119635 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.120640 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:41.122353 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.123219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.124047 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:41.124596 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:41.125326 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:41.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.127092 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:41.269428 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:41.270378 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:41.271259 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:41.271715 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:41.271990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.273072 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.275114 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:41.276239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:45.948467 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:45.950257 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:45.990055 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:45.990675 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:45.991555 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:45.992484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:45.993573 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:46.118841 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:46.120265 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:46.120609 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:46.121188 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.122222 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:46.124168 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.125189 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:46.126083 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:46.126083 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:46.126083 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:46.126083 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.126587 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:46.127870 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.128819 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:46.129409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:46.130291 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:46.130891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:46.130891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.130891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:46.270024 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:46.271076 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:46.271722 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:46.271998 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:46.272166 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.273084 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.274995 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:46.275864 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:50.949027 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:50.950596 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:50.986396 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:50.986891 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:50.987852 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:50.989237 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:50.990218 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:51.119545 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:51.121037 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:51.121461 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:51.121843 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.123018 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:51.124862 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.125679 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:51.126788 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:51.126788 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:51.126788 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:51.127239 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.127724 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:51.128658 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.129219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:51.129990 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:51.130495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:51.131181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:51.131181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.131181 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:51.271178 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:51.271824 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:51.272802 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:51.273291 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:51.273554 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.274590 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.276898 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:51.277719 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:55.949840 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:55.951927 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:55.984980 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:55.985219 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:55.985774 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:55.987190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:55.988172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:56.120393 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:56.122177 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:56.122475 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:56.122747 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.123714 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:56.125504 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.126020 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:56.126513 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:56.126513 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:00:56.127049 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:56.127348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.128074 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:00:56.128828 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.129259 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:00:56.129784 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:56.130220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:56.130742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:56.130742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.130742 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:00:56.272066 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:00:56.272419 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:00:56.273569 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:56.274016 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:00:56.274341 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.275201 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.277312 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:00:56.278090 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:00.950727 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:00.952648 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:00.989535 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:00.989772 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:00.990861 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:00.992172 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:00.993154 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:01.121741 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:01.123831 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:01.124450 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:01.124450 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.125791 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:01.126966 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.127856 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:01.128748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:01.128748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:01:01.128748 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:01.128748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.130107 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:01:01.131467 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.132115 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:01:01.133044 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:01.133685 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:01.134358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:01.134358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.134358 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:01.272789 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:01.273419 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:01.274342 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:01.274818 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:01.275077 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.276052 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.277894 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:01.278793 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:05.952612 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:05.954086 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:05.986327 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:05.986619 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:05.987345 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:05.988360 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:05.989255 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:06.122370 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:06.123923 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:06.124316 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:06.124625 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.125845 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:06.128281 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.129148 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:06.130151 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:06.130151 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:01:06.130625 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:06.131121 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.132220 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:01:06.133429 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.134289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:01:06.135448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:06.136357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:06.137299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:06.137299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.137299 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:06.276758 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:06.277446 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:06.277631 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:06.277852 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:06.277920 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.278409 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.279340 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:06.279788 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:10.952984 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:10.954607 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:10.989079 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:10.989254 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:10.990099 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:10.991050 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:10.991930 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:11.123442 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:11.124961 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:11.125282 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:11.125659 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.126634 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:11.128888 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.129348 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:11.129891 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:11.129891 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:01:11.130160 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:11.130452 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.131335 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:01:11.132713 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.133136 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:01:11.134748 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:11.135184 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:11.135717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:11.135717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.135717 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:11.277275 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:11.278277 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:11.278810 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:11.279220 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:11.279448 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.280424 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.282773 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:11.283538 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:15.953626 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:15.955194 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:15.990081 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:15.990495 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:15.991508 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:15.992661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:15.993526 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:16.124252 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:16.125688 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:16.126054 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:16.126287 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.127383 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:16.129283 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.129887 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:16.130387 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:16.130387 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:01:16.130665 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:16.131030 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.131855 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:01:16.133042 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.133715 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:01:16.134608 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:16.135307 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:16.135801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:16.135801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.135801 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:16.278559 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:16.279132 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:16.279767 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:16.280139 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:16.280368 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.281197 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.283289 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:16.284357 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:20.954281 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:20.955455 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:20.956688 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG dbcounter [-] [105470] Writing DB stats neutron:SELECT=264 {{(pid=105470) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Feb 16 20:01:20.984553 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:20.985018 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:20.985965 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:20.987133 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:20.987591 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:21.124969 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:21.126350 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:21.126622 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:21.127534 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.127768 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:21.129712 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.130484 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:21.131403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:21.131403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:01:21.131403 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:21.131403 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.131837 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:01:21.132735 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.133536 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:01:21.134190 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:21.134908 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:21.135427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:21.135427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.135427 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:21.279058 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:21.279660 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:21.280582 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:21.281064 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:21.281309 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.282229 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.283883 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:21.284807 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:25.954950 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:25.957124 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:25.989803 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:25.990224 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Traceback (most recent call last): Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:25.991129 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-ef4fc7b6-15cf-486c-8c41-df165a1a526f None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 777, in check_fdb_aging_settings Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:25.992019 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:25.993173 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:26.125909 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:26.127119 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b2b0bc6c-3508-4c6e-bdbd-3d979a42abc4, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:26.127394 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-15d28432-bd1b-492c-b98d-eccecccacd15, routes=[], if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:26.127622 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Traceback (most recent call last): Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.128585 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-142fd90d-04e7-4040-852a-25d3c29f103d None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 869, in check_router_default_route_empty_dst_ip Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:26.130217 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.131043 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:26.132012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:26.132012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105470) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1068}} Feb 16 20:01:26.132012 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:26.132012 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Traceback (most recent call last): Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.133324 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Feb 16 20:01:26.134770 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.136378 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.command Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [None req-97da07a4-f5b8-45ee-a7e9-2807905360b7 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1072, in set_fip_distributed_flag Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Feb 16 20:01:26.137191 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^ Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:26.137956 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:26.139027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:26.139027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.139027 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Feb 16 20:01:26.279809 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105470) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:638}} Feb 16 20:01:26.280310 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105470) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:942}} Feb 16 20:01:26.280857 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:26.281297 np0000155677 neutron-ovn-maintenance-worker[105470]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=0) {{(pid=105470) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Feb 16 20:01:26.281390 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: txn.results.put(txn.do_commit()) Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: ^^^^^^^^^^^^^^^ Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: raise RuntimeError(msg) Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.282054 np0000155677 neutron-ovn-maintenance-worker[105470]: Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics [-] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics Traceback (most recent call last): Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics work() Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(*args, **kwargs) Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 82, in decorator Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics return f(self, *args, **kwargs) Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 791, in update_mac_aging_settings Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.283661 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 271, in transaction Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics next(self.gen) Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics with self.create_transaction( Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics self.result = self.commit() Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^ Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise result.ex Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics raise RuntimeError(msg) Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Feb 16 20:01:26.284762 np0000155677 neutron-ovn-maintenance-worker[105470]: ERROR futurist.periodics